Mender install from rescue mode

General Discussions
#1

This is not directly related to mender but thought I could get some info in case people have already done it.
I have a Pi-based system booted in rescue mode. Rescue mode lives in tmpfs completely allowing full control of the underlying storage.
The plan is to provide a USB drive with a full .img file and a rootfs-image mender file. The .img file includes p1=read-only boot, p2 and p3 identical read-only filesystem for active and inactive root and p4 for read-write data storage. The plan is to:

  1. dd write the img file to /dev/mmcblk0 that will create the partitions with default p1, p2, p3 and p4 filesystem
  2. mender install the rootfs-image mender file to p2 and/or p3 to practically push our updated code

I believe after step 1 I would need to mount some locations and chroot to p2 or p3.
Any hints or ideas?

#2

Hi @bender,

If you have an SDIMG file created by either Yocto or mender-convert, then the partitions are already setup with the Mender configurations so the dd step should be sufficient. Am I misunderstanding what you are trying to do?

Drew

#3

This is not an one-off situation. We are trying to create a future-proof process to bring a machine straight out of the factory to an up-to-date state. As we already maintain mender updates, we want to keep a simple small and generic img file (that we will not maintain) and simply get the latest rootfs-image mender file that we produce anyway for other procedures. We also try and do that without any intermediate rebooting.

#4

The problem was that the first dd would erase the area where the environmental variables are stored so mender couldn’t proceed. We copied them from a booted machine and copied them back after the first dd.

mount /dev/sda /mnt
bunzip2 -c /mnt/core-image-base.sdimg.bz2  | dd of=/dev/mmcblk0
dd of=/dev/mmcblk0 bs=1 count=16384 seek=4194304 if=/mnt/envs/uboot_primary.env
dd of=/dev/mmcblk0 bs=1 count=16384 seek=8388608 if=/mnt/envs/uboot_backup.env
umount /mnt
mount -o ro /dev/mmcblk0p3 /mnt
mount -o ro /dev/mmcblk0p1 /mnt/uboot
mount /dev/mmcblk0p4 /mnt/data
mount -o bind /sys/ /mnt/sys/
mount -o bind /proc/ /mnt/proc/
mount -o bind /dev/ /mnt/dev/
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /run/ /mnt/run/
cd /mnt
chroot .
mount -o remount,rw /
mkdir /ttt
mount /dev/sda /ttt
mender install /ttt/core-image-base.mender
dd of=/dev/mmcblk0 bs=1 count=16384 seek=4194304 if=/dev/null
dd of=/dev/mmcblk0 bs=1 count=16384 seek=8388608 if=/dev/null
umount /ttt
exit
cd /
umount /mnt/run/ /mnt/dev/pts /mnt/dev/ /mnt/proc/ /mnt/sys/ /mnt/data /mnt/uboot /mnt

Some parts are specific to our implementation so…

#5

It seems we are not done yet…
Below modified version run in rescue-mode with swupdate manages to:

  • dd an image file directly to mmcblk0
  • mender install a rootfs-image file
  • reboot at the end to /dev/mmcblk0p2

We want to do all that in one go with only one reboot (bullet 3).
The problem is that when booting to normal mode at the end, latest rootfs-image file seems to be installed fine and mender show-artifact reports fine but pushing a differential update just then I get:
ERRO[0000] Artifact dependency "rootfs_image_checksum" not satisfied by currently installed artifact (f808950246263f7fee4f72e96defefa2f0874c46bd6b592925d96ecdf3913bcf != <nil>).

Here is the script called by swupdate:

#!/bin/bash
set -x
MMCBLKFILE="/dev/mmcblk0"
PRIMARYENV="/tmp/uboot_primary.env"
BACKUPENV="/tmp/uboot_backup.env"

systemctl stop mender
dd of=$MMCBLKFILE bs=1 count=16384 seek=4194304 if=$PRIMARYENV
dd of=$MMCBLKFILE bs=1 count=16384 seek=8388608 if=$BACKUPENV
mount -o ro /dev/mmcblk0p3 /mnt
mount -o ro /dev/mmcblk0p1 /mnt/uboot
mount /dev/mmcblk0p4 /mnt/data
mount -o bind /sys/ /mnt/sys/
mount -o bind /proc/ /mnt/proc/
mount -o bind /dev/ /mnt/dev/
mount -o bind /dev/pts /mnt/dev/pts
mount -o bind /run/ /mnt/run/
mount --bind /tmp /mnt/tmp
cd /mnt
chroot . /bin/bash <<"EOT"
mount -o remount,rw /
mender bootstrap &
mender install /tmp/core-image-base.mender
mender commit
dd of=/dev/mmcblk0 bs=1 count=16384 seek=4194304 if=/dev/null
dd of=/dev/mmcblk0 bs=1 count=16384 seek=8388608 if=/dev/null
exit
EOT
cd /
umount /mnt/run/ /mnt/dev/pts /mnt/dev/ /mnt/proc/ /mnt/sys/ /mnt/data /mnt/uboot /mnt/tmp /mnt
reboot

Any hints?

#6

The problem is that you are modifying the root filesystem and then attempting to install a binary delta update. Since the binary delta was generated against a different root filesystem it is unable to apply it. For this to work you will need the first update after running these commands to be a full image.

Drew

#7

It is a full image. The command:

mender install /tmp/core-image-base.mender

is not a differential update but a rootfs-image file. Please re-read my above post. I modified a bit to make it even clearer.

#8

In that case I’m not sure. @kacf does standalone mode set the appropriate provides? Is there another step necessary here?

#9

If you call mender-artifact read /ttt/core-image-base.mender, does it list rootfs_image_checksum in the Provides section?

#10

No it doesn’t! (obscuring some of the info below with ***)

$mender-artifact read core-image-base-***.mender 
Mender artifact:
  Name: base-0.9.5
  Format: mender
  Version: 3
  Signature: signed but no key for verification provided; please use `-k` option for providing verification key
  Compatible devices: '[***]'
  Provides group: 
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   rootfs-image
    Provides: Nothing
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     core-image-base-***.ext4
      size:     1665138688
      modified: 2021-03-19 17:46:50 +0200 EET
      checksum: 52048ec2c3c09191ac2786c005a5e1a3a4aff4ad18ad502250b40cbd77bec791

Why is that? The swu file is generated from Yocto right after Yocto builds the image file and the mender file as usual. Rebuilding as we speak to see it the issue exists in the original mender file too or if the cpio packaging process alters something…

#11

Support for Provides fields was added in mender-artifact 3.3.0. Could you check that the version is at least this version? Also, make sure you have not added --no-checksum-provide anywhere. This is often used to upgrade from older versions of the clients that did not support the Provides fields.

#12

Newly built mender file indeed Provides: Nothing so this is it… Will check our build options.
How come though differentials work if the same artifact (with Provides: Nothing) is installed from the cloud server? Is the Provides added somehow during install?

#13

Well, they shouldn’t, actually. The Provides values are added by default. There are no special options needed to add them, as long as you use the right version and don’t use any options that inhibit them.

#14

My local machine had --no-checksum-provide set while our autobuilder didn’t, hence the confusion. Removing --no-checksum-provide fixed it for good.
Thank you!

#15

Glad you worked it out!