# Mender data partition and read-only operation **URL:** \ **Category:** General Discussions\ **Created:** [February 27, 2020, 11:34pm UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590 "2020-02-27T23:34:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Kris](https://hub.mender.io/u/Kris)\ **Post date:** [February 27, 2020, 11:34pm UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590/1 "2020-02-27T23:34:10Z")
While it is hinted at in the following posts and the manual, nothing ever explicitly says if, when, why or why not the Mender data folder can be set to read-only. > [@Read only compressed rootfs in raw flash?](https://hub.mender.io/t/read-only-compressed-rootfs-in-raw-flash/950): > > On my platform I have NOR flash that is smaller than my rootfs size so would like to use a compressed rootfs that decompresses and runs out of RAM on boot. As far as I can tell, this doesn’t seem to be something that’s supported by Mender out of the box (please correct me if I’m wrong!) so I’ve been looking into how I can achieve this. I wanted to ask if I’m on the right track or if I’m missing anything with my current plan: I’m thinking of using squashfs to compress the rootfs, but am not st… > [@Separation of the partitions for read-only and read-write mode](https://hub.mender.io/t/separation-of-the-partitions-for-read-only-and-read-write-mode/782): > > Hi, I would like to separate the partitions for read-only and read-write mode (Multi-Partitioning). This will reduce the risk of failure. For example ‘/var’, ‘/swap’, ‘/tmp’, ‘/etc/resolv.conf’, etc. can be in read-write mode and the rest of the partitions to be in read-only mode. Is it possible this to be done with Mender and how? Currently, I am interesting for Raspberry Pi 3B+ and Raspberry Pi 4, but this is in general for all embedded, industrial and critical solutions. My question is, once Mender has been bootstrapped on a client, can the data folder ever be read-only or must it always be read-write during operation? With what frequency is data written, and what are the dangers of data corruption here from power loss while read-write. Obviously, if the keys or certs are lost/damaged then all bets are off. But what happens if “mender-store” gets damaged? Additionally, presuming the bare minimum use of the data partition (certs, keys, config, etc.) how much space can Mender be expected to consume of this folder? For reference, I’m asking because we have a small, high-reliability section of flash on a disk that we would like to use for this purpose. However, it requires some hoops to get it mounted read-write on every boot and remain that way. ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 2, 2020, 9:55am UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590/2 "2020-03-02T09:55:26Z")
> [@Kris](#): > > My question is, once Mender has been bootstrapped on a client, can the data folder ever be read-only or must it always be read-write during operation? No, it must always be read-write, because Mender stores its state database here, and this has to survive reboots. > [@Kris](#): > > With what frequency is data written, and what are the dangers of data corruption here from power loss while read-write. Obviously, if the keys or certs are lost/damaged then all bets are off. But what happens if “mender-store” gets damaged? Mender will forget what operation it is running. So for example, if you are in the middle of an update, coming back after a reboot, Mender will act as if it is freshly installed, will neither roll back nor run any state scripts, and will not report the deployment result to the server. > [@Kris](#): > > Additionally, presuming the bare minimum use of the data partition (certs, keys, config, etc.) how much space can Mender be expected to consume of this folder? I don’t have an exact number for you, but in general we’re talking KB. 100K should be more than enough. However, note that Mender uses the same space to provide temporary storage for Update Modules, so if you use any of those you probably want quite a bit more. Most update modules store their payloads here, such as tarballs and packages. This storage is in a subfolder under `/var/lib/mender/modules` (which normally points to `/data/mender/modules`), so you can direct that elsewhere with a symlink if you want. ---
**Author:** ![Kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Kris](https://hub.mender.io/u/Kris)\ **Post date:** [March 2, 2020, 10:41pm UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590/3 "2020-03-02T22:41:31Z")
@kacf Thanks for the information. In looking at the config file, it looks like a handful of things can be stored in different locations based on whats set in mender.conf However, what I don’t see is any specific location to put the client “mender-agent.pem” or “mender-store” ([https://docs.mender.io/2.2/client-configuration/configuration-file/configuration-options](https://docs.mender.io/2.2/client-configuration/configuration-file/configuration-options)). Putting a particularly sensitive file in the same location as a file that presumably gets written somewhat often. The ideal would be to place “mender.conf”, “server.crt”, and “mender-agent.pem” somewhere that is read-only, can be independently updated outside of a Mender artifact image, and not sharing a location with data on a read/write partition that could experience sudden power-off and corruption. Any thoughts on achieving that situation without a setup of symlinks? ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 3, 2020, 9:51am UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590/4 "2020-03-03T09:51:45Z")
`mender-agent.pem` is on the R/W storage so that it can be generated if missing or corrupted. However, note that we have the upcoming feature of [client certificates](https://tracker.mender.io/browse/MEN-3114) that could change this, in this scenario you would configure the location of a client certificate. However, with today’s client I don’t think what you’re asking is possible. Of course I would recommend that you make the R/W filesystem ext4, or some other journalled filesystem, so that corruption is not really possible unless there is a hardware problem. ---
**Author:** ![Kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Kris](https://hub.mender.io/u/Kris)\ **Post date:** [March 5, 2020, 12:07am UTC](https://hub.mender.io/t/mender-data-partition-and-read-only-operation/1590/5 "2020-03-05T00:07:32Z")
“mender-agent.pem is on the R/W storage so that it can be generated if missing or corrupted.” It should never be in a position where it could be corrupted, and there should be a sane, stable way to generate it with a standalone script. This way, one could generate it on first run, save it to somewhere that is mounted read/write at the time, and then will pretty much forever only ever be read-only. Having the same device pop up again to a Mender server because it had to generate a new private key is unexpected and throws red flags. While `mender -bootstrap` will do this generation, it also attempts to contact the remote server. So, even though the private key was generated, `mender -bootstrap` on a non-pre-authorized device will return an error that is indistinguishable from the key failing to be generated. I’ve found that `openssl` can be used to verify that a private key is sane, this seems to work well enough. “Of course I would recommend that you make the R/W filesystem ext4, or some other journalled filesystem, so that corruption is not really possible unless there is a hardware problem.” In the world of users who pull the plug whenever they feel like it, ext4 is not a bulletproof vest. It can help, but its still very possible to end up in a situation with inconsistencies. Compounded with the fact that more devices are moving to solid state drives that end up doing their own internal FTL and wear leveling. Leaving at-risk data on a partition that is mounted read/write is a silly design choice. For what its worth, I’ve got a setup working where `mender-agent.pem`, `mender.conf` (using the /etc location as hard configured .conf, with /data location being able to be field updated for site specific configurations), and `server.crt` are all mounted on a small read-only partition that is also hardware locked to be read-only unless some special knock sequence is used. On first boot, configurations are pulled in from an external device, unformatted partitions are formatted, the above files are copied to the read-only partition with `mender-agent.pem` being generated and copied in if it doesn’t already exist. From that point on, every boot, symlinks are touched on the RW /data partition to point to their RO counterparts before `mender` is started. This ensures that, even if there was some damage, all files can sanely be accessed. `mender` is left to deal with sorting out is state database if it has any inconsistencies. Another bizarre quirk I found is that `mender` wants to remove `mender-agent.pem` rather than just writing to it. This means a pre-existing symlink will be clobbered with the real file. So, the key needs to be generated, relocated, and then a symlink created. I hope this helps anyone else in the same boat as I.