Mender data partition and read-only operation

General Discussions
1

While it is hinted at in the following posts and the manual, nothing ever explicitly says if, when, why or why not the Mender data folder can be set to read-only.

My question is, once Mender has been bootstrapped on a client, can the data folder ever be read-only or must it always be read-write during operation?

With what frequency is data written, and what are the dangers of data corruption here from power loss while read-write. Obviously, if the keys or certs are lost/damaged then all bets are off. But what happens if “mender-store” gets damaged?

Additionally, presuming the bare minimum use of the data partition (certs, keys, config, etc.) how much space can Mender be expected to consume of this folder?

For reference, I’m asking because we have a small, high-reliability section of flash on a disk that we would like to use for this purpose. However, it requires some hoops to get it mounted read-write on every boot and remain that way.

2

No, it must always be read-write, because Mender stores its state database here, and this has to survive reboots.

Mender will forget what operation it is running. So for example, if you are in the middle of an update, coming back after a reboot, Mender will act as if it is freshly installed, will neither roll back nor run any state scripts, and will not report the deployment result to the server.

I don’t have an exact number for you, but in general we’re talking KB. 100K should be more than enough.

However, note that Mender uses the same space to provide temporary storage for Update Modules, so if you use any of those you probably want quite a bit more. Most update modules store their payloads here, such as tarballs and packages. This storage is in a subfolder under /var/lib/mender/modules (which normally points to /data/mender/modules), so you can direct that elsewhere with a symlink if you want.

3

@kacf Thanks for the information.

In looking at the config file, it looks like a handful of things can be stored in different locations based on whats set in mender.conf

However, what I don’t see is any specific location to put the client “mender-agent.pem” or “mender-store” (https://docs.mender.io/2.2/client-configuration/configuration-file/configuration-options). Putting a particularly sensitive file in the same location as a file that presumably gets written somewhat often.

The ideal would be to place “mender.conf”, “server.crt”, and “mender-agent.pem” somewhere that is read-only, can be independently updated outside of a Mender artifact image, and not sharing a location with data on a read/write partition that could experience sudden power-off and corruption.

Any thoughts on achieving that situation without a setup of symlinks?

4

mender-agent.pem is on the R/W storage so that it can be generated if missing or corrupted. However, note that we have the upcoming feature of client certificates that could change this, in this scenario you would configure the location of a client certificate.

However, with today’s client I don’t think what you’re asking is possible. Of course I would recommend that you make the R/W filesystem ext4, or some other journalled filesystem, so that corruption is not really possible unless there is a hardware problem.

5

“mender-agent.pem is on the R/W storage so that it can be generated if missing or corrupted.”
It should never be in a position where it could be corrupted, and there should be a sane, stable way to generate it with a standalone script. This way, one could generate it on first run, save it to somewhere that is mounted read/write at the time, and then will pretty much forever only ever be read-only. Having the same device pop up again to a Mender server because it had to generate a new private key is unexpected and throws red flags.

While mender -bootstrap will do this generation, it also attempts to contact the remote server. So, even though the private key was generated, mender -bootstrap on a non-pre-authorized device will return an error that is indistinguishable from the key failing to be generated. I’ve found that openssl can be used to verify that a private key is sane, this seems to work well enough.

“Of course I would recommend that you make the R/W filesystem ext4, or some other journalled filesystem, so that corruption is not really possible unless there is a hardware problem.”
In the world of users who pull the plug whenever they feel like it, ext4 is not a bulletproof vest. It can help, but its still very possible to end up in a situation with inconsistencies. Compounded with the fact that more devices are moving to solid state drives that end up doing their own internal FTL and wear leveling. Leaving at-risk data on a partition that is mounted read/write is a silly design choice.

For what its worth, I’ve got a setup working where mender-agent.pem, mender.conf (using the /etc location as hard configured .conf, with /data location being able to be field updated for site specific configurations), and server.crt are all mounted on a small read-only partition that is also hardware locked to be read-only unless some special knock sequence is used.

On first boot, configurations are pulled in from an external device, unformatted partitions are formatted, the above files are copied to the read-only partition with mender-agent.pem being generated and copied in if it doesn’t already exist. From that point on, every boot, symlinks are touched on the RW /data partition to point to their RO counterparts before mender is started. This ensures that, even if there was some damage, all files can sanely be accessed. mender is left to deal with sorting out is state database if it has any inconsistencies.

Another bizarre quirk I found is that mender wants to remove mender-agent.pem rather than just writing to it. This means a pre-existing symlink will be clobbered with the real file. So, the key needs to be generated, relocated, and then a symlink created.

I hope this helps anyone else in the same boat as I.