We installed Mender on a Ubuntu 24.04.5 LTS, embedded computer (TQ Group TQMxE41S). The mender-client was fully working, and we are able to see the deployments in Mender as well. We recently installed a new Kernel on Ubuntu (a modified 5.15.0-119 kernel with some small driver patches applied for our embedded module), and since then, intermittently, we have some of our embedded modules in our test devices seem to go from a working state (with Mender running properly) to, on every boot, failing to find the Mender grub files:
In reality, the /grub-mender-grubenv folder is located in (hd2,gpt1)/ not (hd2,gpt2)/ as the system is attempting to find. By default, (hd2,gpt1)/ partition is mounted at /boot/efi and (hd2,gpt2)/ is mounted at /boot (and serves as the prefix). See below:
I’ve attached the mender.conf we use, and the grub.cfg that is generated. We have installed the Mender grub config system and installed it per the instructions on Github.
If you have any suggestions as to why the system is entering this state, or how we can modify the grub behavior to prevent it from happening in the future, please let me know.
Thanks for reaching out and the detailed description. However, a couple of things are not clear to me yet:
“installed on Ubuntu”: does this mean you installed the Client through apt or whatever from inside the running system, or did you use mender-convert?
you said “installed a new kernel”: does this mean the Ubuntu package process?
When you say “some” modules, is there any common characteristic? Like all which are on active partition A, or B?
My guess is that the kernel installation process regenerated parts of the grub configuration, overwriting something that ties into the Mender integration. Reproducing that with a before-after dump of the grub configuration directory.
As a side note, having the data partition before the A/B ones means that you can’t grow it, hence we recommend it to be the last partition.
Thank you for the detailed response, here are the answer to your questions:
“installed on Ubuntu”: does this mean you installed the Client through apt or whatever from inside the running system, or did you use mender-convert? Installed through apt using the mender docs recommended process, for Ubuntu 22.04
you said “installed a new kernel”: does this mean the Ubuntu package process? Yes, we use the command dpkg -i linux-BeamlinkPatch-kernel119.deb to install the new kernel before starting the apt installation of the mender client
When you say “some” modules, is there any common characteristic? Like all which are on active partition A, or B? All modules are active on Partition A and are identical SKU. All have identical BIOS and configurations loaded from the factory.
I understand your suggestion that the kernel installation could have corrupted something. However, I should have made it clearer in my original post, that we install the kernel patch on top of the default Ubuntu kernel before starting the Mender installation process using apt. There isn’t a before/after set of configuration files because the kernel is installed before mender.
This variable, mender_ptable_part is defined from:
if [ "${mender_boot_part}" = "${mender_rootfsa_part}" -a test -n "${mender_kernela_part}" ]; then
mender_ptable_part=${mender_kernela_part}
mender_kernel_path=""
elif [ "${mender_boot_part}" = "${mender_rootfsb_part}" -a test -n "${mender_kernelb_part}" ]; then
mender_ptable_part=${mender_kernelb_part}
mender_kernel_path=""
else
mender_ptable_part=${mender_boot_part}
mender_kernel_path="/boot"
fi
Therefore, I would expect the mender_ptable_partpartition to be set to either “4” or “5”.
However, the actual grub-mender-grubenv/ files are located, for us, in partition “1”, which is mounted at /boot/efi.
Even more confusingly, in the actual boot messages, it turns out that the partition it is searching for is (hd2,gpt2) which is not partition 1 (where the actual grub environment files are located) and also not partition 4 or 5 that I would expect based on the grub.cfg.
How does this mechanism in the GRUB GNU bootloader system work for choosing the partition for the GRUB mender environment?
