I’ve discovered the new split Mender config when updating from 1.5 to 2.1. It seems to me that the config is split backward of the way it should be done.
The /etc config file part of the rootfs. It’s in the mender image. It’s updated when new firmware is built and installed via mender. If it’s changed and update is applied, the changes are lost and replaced with that’s in the new firmware. If changes should be made to it because of changes in the firmware, an update will do this.
The /data config file is persistent across updates. It doesn’t change after a Mender update. Changes made to it will not be lost after an update. If changes need to be made to it because of a change in the firmware, an update will not do this.
So supposed I want to configure the mender server address. If I put this in /etc, then the next update will revert it back to the default. Isn’t it a lot more likely that the server address shouldn’t change across updates? Instead of being baked into the firmware. I’ve yet to order a “bespoke router”, where they need your domainname at the factory to make the firmware.
Shouldn’t any configuration I want to change, like the specific network location of a device, be stored with persistent data, in /data? Not baked into the firmware to be reset each time the firmware is updated?
For the RootfsPartA/B configuration options, why would I change those? If the hardware changed, but then isn’t the mender update for specific hardware? Or maybe if the kernel, device-tree, or udev config changed in such a way as to change the device name. But all that is part of the rootfs. So if I do change the device tree and need to update RootfsPartA, then I’d want that to be in /etc so it would get updated to match the device tree.
Why was the config split done this way?
It seems like maybe the idea was that /data doesn’t change while /etc is for local config changes. But that’s not really how it works on embedded devices. Ideally /etc should part of a read-only rootfs and never change. Configuration changes need to go to /data so they persist across firmware updates.
It’s the /etc/mender/mender.conf file which gets updated every time you deploy something with Mender, so naturally that’s where the majority of the config entries should go, at least the ones you would want to change. Why do you want to make potentially dangerous changes to /data/mender/mender.conf without going through the robust update mechanism?
But entire rootfs is deployed. This means in order to do something like change what server URL should be used it’s necessary to build an entire custom firmware image. Why should changing the server URL require one have the entire source to the firmware and do a Yocto build of it?
And then why have the partition numbers in /data? That’s even more potentially dangerous since it also breaks a manual install.
The /data/mender/mender.conf file is not meant to ever change, so it can be considered read-only. This is why it has the partition numbers, which will not change once the device is deployed. All other options are kept in /etc/mender/mender.conf so they can be changed by updating the rootfs. The idea is that the maximum number of configuration settings should be on the rootfs, so that they can be updated using a new Mender Artifact. The only exceptions are things that will never change (partition numbers, etc), or things that must survive an update (Mender DB and private auth key).
Mender doesn’t mandate what you keep in each config file, so you are free to change it if it doesn’t suit you.
There is MENDER_PERSISTENT_CONFIGURATION_VARS, and I’ve used it to split the config in the opposite manner, but there are limitations. For one, it doesn’t work on TenantToken. But the big one is that it doesn’t change the precedence order.
So you can not specify local persistent configuration in /data and then fall-back to the rootfs and use /etc (or /lib, e.g. systemd’s search order) if the value isn’t found in /data. It only works the other way around, /etc first then /data.
This is a reason updating from Mender 1.5 (pre split) to Mender 2.1 (post split) doesn’t work. And there isn’t a way to fix the configuration via Mender(*) because Mender doesn’t update the persistent data. If it was possible to put the RootfsPart variables in a rootfs config file and use them iff they aren’t present in /data, it wouldn’t have been a problem.
Also consider the nature of the partitions and filesystems. You have one partition which is mounted read-only and is never modified, it shouldn’t be possible to do so, by the running software. Another partition is read-write and is modified constantly by anything that needs to save non-volatile data. You’ve got a config file with vital information that should never change. Which partition is it going to be safer in, the first or the second?
(*) I know about the updater script, but it doesn’t work unless you happened to have already included jq in the existing rootfs you want to update from.
What triggered the current implementation was that we saw devices getting bricked by updates that had the wrong configuration in the rootfs. Normally Mender would have rolled back, but when the partition information is wrong it can do serious damage instead. It sounds like it should never happen in practice, but this was triggered by real events. For the U-Boot configuration it is even more common, and more damaging.
I agree that the separation of constant data and volatile data could be better in Mender though, and ideally a “constant” partition would have been preferable. For your use case, might I suggest that you point /var/lib/mender, which normally points to /data/mender, to a read-only directory instead. And from within there, symlink the database file to a read-write partition? I haven’t tested that it works, but worth a shot to get the best out of what is there today.