Active root partition matches neither RootfsPartA nor RootfsPartB

Yury · April 4, 2019, 8:26pm

Hello,

I am trying to bringup mender client on yocto poky-tiny based distro. Mender compiled OK, but update is not working. According to debug printouts it is not able to get active rootfs. It look like mender takes active root fs from mount command but in my case it show /dev/root. I think that mender expect name /dev/mmcblk0p2 but not /dev/root. Do you have ideas what could be wrong here ?

root@pynq-z1:/run/media/sda1# mender -debug --rootfs test-image-pynq-z1.mender·
DEBU[0000] Reading Mender configuration from file /etc/mender/mender.conf  module=config
DEBU[0000] Read data from device manifest file: device_type=pynq-z1  module=mender
DEBU[0000] Found needed line: device_type=pynq-z1        module=mender
DEBU[0000] Current manifest data: pynq-z1                module=mender
DEBU[0000] Starting device update.                       module=rootfs
INFO[0000] Start updating from local image file: [test-image-pynq-z1.mender]  module=rootfs
Jan  1 00:17:26 (none) kern.info mender[869]: level=info msg="Start updating from local image file: [test-image-pynq-z1.mender]" module=rootfs·
DEBU[0000] Fetching update from file results: [&{0x10760840}], 23140864, <nil>  module=rootfs
Installing update from the artifact of size 23140864
DEBU[0000] checking if device [pynq-z1] is on compatibile device list: [pynq-z1]
  module=installer
DEBU[0000] installing update test-image-pynq-z1.ext4 of size 973078528  module=installer
DEBU[0000] Trying to install update of size: 973078528   module=device
DEBU[0000] Have U-Boot variable: mender_boot_part=2      module=bootenv
DEBU[0000] List of U-Boot variables:map[mender_boot_part:2]  module=bootenv
DEBU[0000] Setting active partition from mount candidate: /dev/root  module=partitions
ERRO[0000] update image installation failed: Active root partition matches neither RootfsPartA nor RootfsPartB.  module=installer
Jan  1 00:17:26 (none) kern.err mender[869]: level=error msg="update image installation failed: Active root partition matches neither RootfsPartA nor RootfsPartB." module=installer·
Jan  1 00:17:26 (none) kern.err mender[869]: level=error msg="Installation failed: installer: failed to read and install update: update: can not install update: &{test-image-pynq-z1.ext4 420 1000 1000 973078528 2019-04-04 18:02:18 +0:
ERRO[0000] Installation failed: installer: failed to read and install update: update: can not install update: &{test-image-pynq-z1.ext4 420 1000 1000 973078528 2019-04-04 18:02:18 +0000 UTC 48    0 0 0001-01-01 00:00:00 +0000 UTC 000s
ERRO[0000] installer: failed to read and install update: update: can not install update: &{test-image-pynq-z1.ext4 420 1000 1000 973078528 2019-04-04 18:02:18 +0000 UTC 48    0 0 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +000n
root@pynq-z1:/run/media/sda1#·
root@pynq-z1:/run/media/sda1#·
root@pynq-z1:/run/media/sda1# mount
/dev/root on / type ext4 (rw,relatime,data=ordered)
/dev/root on / type ext4 (rw,relatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=246532k,nr_inodes=61633,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/volatile type tmpfs (rw,relatime)
/dev/mmcblk0p1 on /uboot type vfat (rw,sync,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p4 on /data type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /run/media/sda1 type vfat (rw,relatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

mirzak · April 4, 2019, 8:41pm

/dev/root should not be a problem as it checks block device id among other things.

Have you gone trough the integration checklist?

https://docs.mender.io/1.7/devices/yocto-project/bootloader-support/u-boot/integration-checklist

This usually will pickup problems with the integration.

Yury · April 4, 2019, 9:08pm

yes, did this. All checks are passed.

For me the following code looks strange:

github.com

mendersoftware/mender/blob/master/partitions.go#L244


func maybeResolveLink(unresolvedPath string) string {
	// If the supplied path is not a link the original path is returned
	resolvedPath, err := filepath.EvalSymlinks(unresolvedPath)
	// This would only happen if supplied a link that goes nowhere or creates a loop
	if err != nil {
		log.Warnf("Could not resolve path link: %s Attempting to continue", unresolvedPath)
		return unresolvedPath
	}
	// MEN-2302
	// Only resolve /dev/disk/by-partuuid/
	if path.Dir(unresolvedPath) == "/dev/disk/by-partuuid" {
		return resolvedPath
	}
	return unresolvedPath
}

In my case mender found active partition by mount command:
DEBU[0000] Setting active partition from mount candidate: /dev/root module=partitions

Therefore maybeResolveLink function is comparing

if path.Dir(/dev/root) == "/dev/disk/by-partuuid" {
	return resolvedPath

}

Which never returns resolvedPath and as result mender prints problem ErrorPartitionNoMatchActive. Or probably I missed something ?

mirzak · April 4, 2019, 9:24pm

Hm.

Forgot to ask which version you are running and also can you share /etc/mender/mender.conf

Yury · April 4, 2019, 9:31pm

I am using yocto meta layers 2018.3 and therefore i am using mender meta from tag rocko-v2018.03
it fetching mender version mender/1.4.0-r0

This link is correct https://github.com/mendersoftware/mender/blob/1.4.x/partitions.go#L72
In my case active = /dev/root. This condition will always go to else.

root@pynq-z1:~# cat /etc/mender/mender.conf
{
“InventoryPollIntervalSeconds”: 1800,
“RetryPollIntervalSeconds”: 300,
“RootfsPartA”: “/dev/mmcblk0p2”,
“RootfsPartB”: “/dev/mmcblk0p3”,
“ServerURL”: “https://my-mender-server.net/”,
“TenantToken”: “dummy”,
“UpdatePollIntervalSeconds”: 1800
}root@pynq-z1:~#

mirzak · April 4, 2019, 9:35pm

Oki that is a very old release, could you try tip of rocko branch which should build 1.7.0 mender client where I would not expect these type of problems.

mirzak · April 4, 2019, 9:39pm

Also the linked code is not where it fails, it is this function that fails,

github.com

mendersoftware/mender/blob/1.4.x/partitions.go#L150


	devices []string, root *syscall.Stat_t) (string, error) {


	for _, device := range devices {
		if rootChecker(sc, device, root) {
			return device, nil
		}
	}
	return "", RootPartitionDoesNotMatchMount
}


func (p *partitions) getAndCacheActivePartition(rootChecker func(StatCommander, string, *syscall.Stat_t) bool,
	getMountedDevices func(string) ([]string, error)) (string, error) {
	mountData, err := p.Command("mount").Output()
	if err != nil {
		return "", err
	}


	mountCandidate := getRootCandidateFromMount(mountData)
	rootDevice := getRootDevice(p)
	if rootDevice == nil {
		return "", errors.New("Can not find root device")

mirzak · April 4, 2019, 9:40pm

Also looking at the log again, you seem to have two entries for / in result of “mount”, which is a bit odd?

Yury · April 4, 2019, 9:49pm

Yes, i also noticed this. But root is passed only in kernel cmd line. No any mount entries in fstab or other scripts…

console=ttyPS0,115200 root=/dev/mmcblk0p2 rootwait earlyprintk devtmpfs.mount=1 uio_pdrv_genirq.of_id=generic-uio

mirzak · April 4, 2019, 10:07pm

Checking the 1.4.x code this should work, because what the client essentially does is calling (output from my device)

 $ stat /
  File: /
  Size: 1024            Blocks: 2          IO Block: 1024   directory
Device: b302h/45826d    Inode: 2           Links: 20
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-04-04 21:54:41.000000000
Modify: 2019-04-02 11:40:29.000000000
Change: 2019-04-02 11:40:29.000000000

and compares the “Device” field with major minor numbers of following,

$ ls -alh /dev/mmcblk0p2 
brw-rw----    1 root     disk      179,   2 Apr  4 21:54 /dev/mmcblk0p2

And in my case b302 represents 179 Major and 2 minor numbers. To decode it,

#define major(dev) ((int)(((unsigned int) (dev) >> 8) & 0xff))
#define minor(dev) ((int)((dev) & 0xff))

You could manually check the numbers of the output of mount and I suspect this has something to do with the double entries.

Yury · April 4, 2019, 10:34pm

According to the 1.4 code, client using output of mount command to get root device. In my case it gets /dev/root.

Here variable with root device filled https://github.com/mendersoftware/mender/blob/1.4.x/partitions.go#L67
GetActive() -> getAndCacheActivePartition()-> this returns /dev/root (clarified by debug log)

In this case condition https://github.com/mendersoftware/mender/blob/1.4.x/partitions.go#L72 never be true, therefore it triggers error https://github.com/mendersoftware/mender/blob/1.4.x/partitions.go#L77

Regarding root mounted twice, it looks like mount command show incorrect output. I am using musl c and busybox commands to make the system as much as possible small.

If i run “cat /proc/mounts” it show only one mount root

root@pynq-z1:~# mount 
/dev/root on / type ext4 (rw,relatime,data=ordered)                                                    
/dev/root on / type ext4 (rw,relatime,data=ordered)                                                                               devtmpfs on /dev type devtmpfs (rw,relatime,size=246532k,nr_inodes=61633,mode=755)                     
proc on /proc type proc (rw,relatime)                                                                 
sysfs on /sys type sysfs (rw,relatime)                                                                 
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /var/volatile type tmpfs (rw,relatime)
/dev/mmcblk0p1 on /uboot type vfat (rw,sync,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p4 on /data type ext4 (rw,relatime,data=ordered)
root@pynq-z1:~# 
root@pynq-z1:~# 
root@pynq-z1:~# 
root@pynq-z1:~# cat /proc/mounts 
/dev/root / ext4 rw,relatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=246532k,nr_inodes=61633,mode=755 0 0
proc /proc proc rw,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /var/volatile tmpfs rw,relatime 0 0
/dev/mmcblk0p1 /uboot vfat rw,sync,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
/dev/mmcblk0p4 /data ext4 rw,relatime,data=ordered 0 0
root@pynq-z1:~# 
root@pynq-z1:~# 
root@pynq-z1:~# strace mount
execve("/bin/mount", ["mount"], 0xbe82ce50 /* 10 vars */) = 0
set_tls(0xb6f3e0ec, 0, 0x490c1f0c, 0x1, 0xb6f3e044) = 0
set_tid_address(0xb6f3e060)             = 773
mprotect(0xa7000, 4096, PROT_READ)      = 0
getuid32()                              = 0
getuid32()                              = 0
geteuid32()                             = 0
open("/proc/mounts", O_RDONLY|O_LARGEFILE) = 3
readv(3, [{iov_base="", iov_len=0}, {iov_base="/dev/root / ext4 rw,relatime,dat"..., iov_len=1024}], 2) = 575
ioctl(1, TIOCGWINSZ, {ws_row=67, ws_col=237, ws_xpixel=0, ws_ypixel=0}) = 0
writev(1, [{iov_base="/dev/root on / type ext4 (rw,rel"..., iov_len=50}, {iov_base=")\n", iov_len=2}], 2/dev/root on / type ext4 (rw,relatime,data=ordered)
) = 52
writev(1, [{iov_base="/dev/root on / type ext4 (rw,rel"..., iov_len=50}, {iov_base=")\n", iov_len=2}], 2/dev/root on / type ext4 (rw,relatime,data=ordered)
) = 52
writev(1, [{iov_base="devtmpfs on /dev type devtmpfs ("..., iov_len=81}, {iov_base=")\n", iov_len=2}], 2devtmpfs on /dev type devtmpfs (rw,relatime,size=246532k,nr_inodes=61633,mode=755)
) = 83
writev(1, [{iov_base="proc on /proc type proc (rw,rela"..., iov_len=36}, {iov_base=")\n", iov_len=2}], 2proc on /proc type proc (rw,relatime)
) = 38
writev(1, [{iov_base="sysfs on /sys type sysfs (rw,rel"..., iov_len=37}, {iov_base=")\n", iov_len=2}], 2sysfs on /sys type sysfs (rw,relatime)
) = 39
writev(1, [{iov_base="devpts on /dev/pts type devpts ("..., iov_len=65}, {iov_base=")\n", iov_len=2}], 2devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
) = 67
writev(1, [{iov_base="tmpfs on /run type tmpfs (rw,nos"..., iov_len=50}, {iov_base=")\n", iov_len=2}], 2tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
) = 52
writev(1, [{iov_base="tmpfs on /var/volatile type tmpf"..., iov_len=46}, {iov_base=")\n", iov_len=2}], 2tmpfs on /var/volatile type tmpfs (rw,relatime)
) = 48
writev(1, [{iov_base="/dev/mmcblk0p1 on /uboot type vf"..., iov_len=141}, {iov_base=")\n", iov_len=2}], 2/dev/mmcblk0p1 on /uboot type vfat (rw,sync,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=)
) = 143
writev(1, [{iov_base="/dev/mmcblk0p4 on /data type ext"..., iov_len=59}, {iov_base=")\n", iov_len=2}], 2/dev/mmcblk0p4 on /data type ext4 (rw,relatime,data=ordered)
) = 61
readv(3, [{iov_base="", iov_len=0}, {iov_base="", iov_len=1024}], 2) = 0
exit_group(0)                           = ?
+++ exited with 0 +++
root@pynq-z1:~#

Yury · April 4, 2019, 11:05pm

I checked version 1.7.x and it is able to find root partition because there is added function to resolve /dev/root symlink. https://github.com/mendersoftware/mender/blob/e3eda1307476eb8992d45b0b1cc7d926a7cce1b2/partitions.go#L234

I had not checked on master but it seems problem exists. There is check which never be true if we pass /dev/root https://github.com/mendersoftware/mender/blob/master/partitions.go#L244

wjax · November 2, 2021, 3:55pm

Hi there.

I am having the same problem with a custom integration.
Active partition is not recognized

DEBU[0000]/root/work/build/debos_mgw221_squidsmartapp_3.0.24/build/mender-3.1.0/_gopath/src/github.com/mendersoftware/mender/installer/partitions.go:55 github.com/mendersoftware/mender/installer.(*partitions).GetActive() Active partition: /dev/root
ERRO[0000]/root/work/build/debos_mgw221_squidsmartapp_3.0.24/build/mender-3.1.0/_gopath/src/github.com/mendersoftware/mender/app/standalone.go:166 github.com/mendersoftware/mender/app.doStandaloneInstallStatesDownload() Download failed: Payload: can not install Payload: rootfs.ext4: Active root partition matches neither RootfsPartA nor RootfsPartB.

mender client 3.1.0

wjax · November 3, 2021, 7:54pm

I have managed to solve the problem.

In my system (and maybe in others), /dev/root is the mounted on /.
However /dev/root is a symlink to the underlying real particion “mmc2blk2”

So getAndCacheActivePartition does not follow the link and keeps trying to use /dev/root and fails (obviously).

So what I have done is check that whatever returns mount is not a symlink, if it is, guess what the real path is.

        // First check if mountCandidate matches rootDevice
	if mountCandidate != "" {

		// Resolve link
		resolvedLink := maybeResolveLink(mountCandidate)
		if strings.Contains(p.rootfsPartA, resolvedLink) {
			resolvedLink = p.rootfsPartA
		} else if strings.Contains(p.rootfsPartB, resolvedLink) {
			resolvedLink = p.rootfsPartB
		}

		if rootChecker(p, resolvedLink, rootDevice) {
			p.active = resolvedLink
			log.Debugf("Setting active partition from mount candidate: %s", p.active)
			return p.active, nil
		}

		// If mount candidate does not match root device check if we have a match in ENV
		if checkBootEnvAndRootPartitionMatch(bootEnvBootPart, mountCandidate) {
			p.active = mountCandidate
			log.Debug("Setting active partition: ", mountCandidate)
			return p.active, nil
		}
		// If not see if we are lucky somewhere else
	}

I have also modified maybeResolveLink to always resolv and not only /dev/disk/by-partuuid/

This is not the best implementation but it solves the problem.

Do you “mender guys” want me to send a pull request? If so, can you guide me on what solution you like more?

Regards

kacf · November 4, 2021, 7:49am

As mentioned in the ticket reference in the code, we cannot resolve all links because it breaks LVM. However, I think we can quite safely, and reasonably, resolve /dev/root specifically, since this is a common thing that mount returns.

If you could send a pull request for this, it would be great!

wjax · November 5, 2021, 8:29am

Done

Resolv symlink /dev/root while getting active partition by wjax · Pull Request #876 · mendersoftware/mender (github.com)

Topic		Replies	Views
Problem with rootfs update with *.mender General Discussions	18	1274	August 10, 2021
No match between boot and root partitions General Discussions	11	961	August 13, 2021
Updating when rootfs is in RAM? General Discussions	6	903	October 11, 2019
/dev/root mounted on / is FULL General Discussions	10	1662	June 22, 2020
How to deploy /data/mender on rootfs General Discussions yocto , dunfell	1	500	January 5, 2022

Active root partition matches neither RootfsPartA nor RootfsPartB

Related Topics