Single File Update - Not overwriting old file

General Discussions
1

Hi,

I have a couple of RPi4’s all setup with hosted mender under the basic plan, each of these units unpacks a zipped folder located in the home directory on boot and runs the program inside

To update this program I’ve been trying to upload the new zipped folder to Mender under the Releases tab and creating a new single file update with the target directory set to the home directory (e.g. /home/pi)

I can upload the file and create a new deployment, rebooting the Pi’s they pick this up and Mender shows them all downloading/installing

However on reboot I’ll find that only a handful have actually updated to the new version of the program even though the Deployment lists all as successful

The folder always has the same name and I’d expect it to just overwrite any existing folder - is there something I’m missing here?

Many thanks

2

I can’t immediately see any wrong steps. Does it work if you install it in standalone mode with mender install <ARTIFACT>?

3

Thanks for the reply - will give that a try when I next have access to the units

I have been informed one of the units is reporting a segmentation fault, so it sounds like it may be a problem elsewhere (memory card itself?), will update once I’ve done some further testing

4

I’ve tried it on a freshly setup board and it still seems to be skipping the overwrite

Perhaps its because the file is ‘in use’ (being unpacked) whilst mender is trying to update it just after boot?

If I instead upload the file directly via Mender-Connect after the old file has been unpacked there’s no problem

I’ve read that rootfs updates write to the 2nd partition and then swap over on reboot, is it possible to do that with single files? Or is there a way to tell Mender to hold off on updates until ready?

Otherwise I may need to make my startup sequence a bit more intelligent

5

There are several ways you can accomplish this. If it’s only happening at boot, I would consider using standard systemd unit ordering. That is, in the service which runs your program, put:

[Unit]
Before=mender-client.service

See systemd documentation for more information.

If it can also happen at arbitrary times, consider using an ArtifactInstall_Enter state script which blocks if it detects that a conflicting process is running.

6

Thanks for the reply, I’ve tried moving the script to systemd, the device now checks in to Mender after the program is running which is all good

However still seeing the same issue when trying to push a single file update

I reread your initial response and tried copying the artifact onto the Pi and running manually, and I’m shown a error about there not being enough space which is perhaps more helpful, see snippet below

alex@gateway-dev-01:~ $ sudo mender install artifact.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p2  
INFO[0000] Start updating from local image file: [artifact.mender] 
Installing Artifact of size 72000000...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0004] Update module output: cp:                    
INFO[0004] Update module output: error writing '/var/lib/mender/modules/v3/payloads/0000/tree/tmp/dest_dir.tmp' 
INFO[0005] Update module output: : No space left on device 
INFO[0005] Update module output:                        
Use -commit to update, or -rollback to roll back the update.

If I delete the file that’s being replaced then there’s no issue

alex@gateway-dev-01:~ $ rm program.enc 
alex@gateway-dev-01:~ $ sudo mender install artifact.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p2  
INFO[0000] Start updating from local image file: [artifact.mender] 
Installing Artifact of size 72000000...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
Use -commit to update, or -rollback to roll back the update.

Would you have any ideas?

df below if its any help, I may have misunderstood something during the mender-convert process and not provided enough extra space, this is basically a converted stock raspi os image on a 16gb card

alex@gateway-dev-01:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        2016600 1689748    206364  90% /
devtmpfs         1827468       0   1827468   0% /dev
tmpfs            1959564       0   1959564   0% /dev/shm
tmpfs            1959564    8628   1950936   1% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs            1959564       0   1959564   0% /sys/fs/cgroup
/dev/mmcblk0p4    110576   70532     30872  70% /data
/dev/mmcblk0p1    258095   43911    214184  18% /uboot
tmpfs             391912       0    391912   0% /run/user/1001
7

I don’t fully grasp what’s going on, to be honest… Can you add set -x after set -e in the /usr/share/mender/modules/v3/single-file Update Module script? That will add debug output which might make it easier to see what’s happening.

8

Thanks, I’ve given that a try, output below although looks pretty much the same:

alex@gateway-dev-01:~ $ sudo mender install artifact.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p2  
INFO[0000] Start updating from local image file: [artifact.mender] 
Installing Artifact of size 72000000...
INFO[0000] No public key was provided for authenticating the artifact 
ERRO[0000] Installer: Error initializing directory for scripts [/var/lib/mender/scripts]: mkdir /var/lib/mender/scripts: no space left on device 
ERRO[0000] Reading headers failed: installer: error initializing directory for scripts: mkdir /var/lib/mender/scripts: no space left on device 
ERRO[0000] installer: error initializing directory for scripts: mkdir /var/lib/mender/scripts: no space left on device

I’ve noticed that the directory its complaining about /var/lib/mender seems to link to /data

And data only seems to have 128MB of space, could it be Mender is copying the update there but finding there’s not enough space so has to cancel?

alex@gateway-dev-01:~ $ ls -l /var/lib
.....
lrwxrwxrwx 1 root    root      12 Jul  6 11:02 mender -> /data/mender
lrwxrwxrwx 1 root    root      22 Jul  6 11:02 mender-configure -> /data/mender-configure
...
9

Yes, the /var/lib/mender folder needs to point to the data partition. This is part of a normal dual rootfs partition setup.

But the data partition is supposed to grow to fill the available space. What kind of image are you using? Downloaded from mender.io? Mender-convert? Yocto?

If possible, try to adjust MENDER_DATA_PART_SIZE_MB during creation, which will shrink the rootfs partitions in order to make the data partition bigger.

10

It’s plain old Raspberry Pi OS converted using mender-convert, taken earlier this year so would still be Debian Buster

Had a bit of a play with it over the weekend, I did try manually resizing the data partition via gparted and that seemed to fix the issue, I’ll next try specifying partition sizes in the config and running mender-convert again to make a new image as I’m being left with 8GB of unused space by default!

I think the reason its not happening automatically is due to the systemd version, can’t find it now but I’m sure I remember seeing a requirement for specific version and when checking I was a version behind