Apt or dpkg install causing system to break

We have an embedded system running ont he Raspberry Pi 4 and CM4 modules. We have succesfully implemented mender to manage updates via artifacts.

Our application is apcakged as a debian package which allows us to install the application via our own apt repository or dpkg installation locally. This installation path is used for development purposes only, so it does not cause issues for our deployed devices.

When installating via apt or dpkg this causes the system to brick entierly most of the time (sometimes it succeeds without issue). The installation proceeds succesfully, but once the system is rebooted then we don’t even get any boot outputs/kernel messages. I suspect the issue is similar to the one reported at Raspbian update breaks devices | Mender . I suspect our package installation is causing a partition or boot issue, but I am unsure where to look. Do you have any suggestions at how best to debug this issue? We can mount the SD card on a recovery system is needed.

This is a known issue. Due to how the bootloader is set up on Raspberry Pi, running apt is not safe there. There are plans to fix it, but the fix is quite substantial, and will likely mean switching to a different bootloader altogether, so I’m not sure when that will happen.

The recommended flow is to make modifications on a pristine system without Mender, and then use mender-convert to make an update out of that afterwards. On the production devices, only rootfs updates should be deployed.

Thank you for the quick reply and recommended procedure.