Remounting root FS takes far too long

General Discussions
1

Hey there,

on our device the root-fs is re-mounted after “a long period of time” (~12 secs), but I don’t know whats going on between timestamp [ 1.438409] (first mount) and [ 13.173064] (re-mount). Why is there a re-mount to “ro” again? What’s going on after timestamp [ 4.873242] where the boot process does nothing?

Has someone a hint?

System:
Yocto Scathgap, U-Boot 2024.07, RT-Kernel 6.6.65, Mender-Client 4.0.7, SysV
Yocto: IMAGE_FEATURES += “read-only-rootfs”
SysVinit: ROOTFS_READ_ONLY=yes

dmesg-output:

[...]
[    0.330407] mmc0: SDHCI controller on 2198000.mmc [2198000.mmc] using ADMA
[    0.337793] NET: Registered PF_PACKET protocol family
[    0.337831] Key type dns_resolver registered
[    0.340626] Registering SWP/SWPB emulation handler
[    0.349007] Loading compiled-in X.509 certificates
[    0.390644] mmc0: new DDR MMC card at address 0001
[    0.391787] mmcblk0: mmc0:0001 Q2J54A 3.59 GiB
[    0.396169] pps pps0: new PPS source ptp0
[    0.398197]  mmcblk0: p1 p2 p3 p4 < p5 p6 >
[    0.399929] mmcblk0boot0: mmc0:0001 Q2J54A 16.0 MiB
[    0.401931] mmcblk0boot1: mmc0:0001 Q2J54A 16.0 MiB
[    0.404567] mmcblk0rpmb: mmc0:0001 Q2J54A 512 KiB, chardev (248:0)
[    0.421728] fec 2188000.ethernet eth0: registered PHC device 0
[    0.422441] imx_thermal 20c8000.anatop:tempmon: Industrial CPU temperature grade - max:105C critical:105C passive:95C
[    0.425207] sdhci-esdhc-imx 2190000.mmc: Got CD GPIO
[    0.426664] clk: Disabling unused clocks
[    0.459874] mmc1: SDHCI controller on 2190000.mmc [2190000.mmc] using ADMA
[    1.438409] EXT4-fs (mmcblk0p3): mounted filesystem 7886976b-c57d-43b2-aa41-04860776fe30 ro without journal. Quota mode: disabled.
[    1.438491] VFS: Mounted root (ext4 filesystem) readonly on device 179:3.
[    1.439164] devtmpfs: mounted
[    1.440362] Freeing unused kernel image (initmem) memory: 1024K
[    1.440633] Run /sbin/init as init process
[    1.440641]   with arguments:
[    1.440648]     /sbin/init
[    1.440653]   with environment:
[    1.440658]     HOME=/
[    1.440664]     TERM=linux
[    1.440669]     enable_wait_mode=off
[    1.440674]     fec_mac=00:14:2d:a4:12:4f
[    1.440680]     consoleblank=0
[    1.440684]     fbmem=8M
[    1.785599] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[    1.787053] EXT4-fs: Warning: mounting with data=journal disables delayed allocation, dioread_nolock, O_DIRECT and fast_commit support!
[    1.944152] EXT4-fs (mmcblk0p5): recovery complete
[    1.945279] EXT4-fs (mmcblk0p5): mounted filesystem f75abc7b-3ecd-43c8-b7dd-96c4bb6d74aa r/w with journalled data mode. Quota mode: disabled.
[    1.950232] EXT4-fs (mmcblk0p6): mounted filesystem 1c15fdee-4e52-49b6-bd6c-16608aea381f ro with ordered data mode. Quota mode: disabled.
[    2.093949] udevd[122]: starting version 3.2.14
[    2.129547] udevd[123]: starting eudev-3.2.14
[    2.278259] CAN device driver interface
[    2.288416] spi_master spi1: will run message pump with realtime priority
[    2.313605] spi_master spi3: will run message pump with realtime priority
[    2.449548] imx-sdma 20ec000.dma-controller: Direct firmware load for imx/sdma/sdma-imx6q.bin failed with error -2
[    2.449598] imx-sdma 20ec000.dma-controller: external firmware not found, using ROM firmware
[    2.470790] usb_phy_generic usbphynop1: dummy supplies not allowed for exclusive requests
[    2.473050] usb_phy_generic usbphynop2: dummy supplies not allowed for exclusive requests
[    2.790178] Micrel KSZ8041 2188000.ethernet-1:00: attached PHY driver 
(mii_bus:phy_addr=2188000.ethernet-1:00, irq=POLL)
[...]
[    3.460711] /dev/mmcblk0p1: Can't open blockdev
[    3.460816] /dev/mmcblk0p1: Can't open blockdev
[    3.460891] /dev/mmcblk0p1: Can't open blockdev
[    4.873242] fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[   13.173064] EXT4-fs (mmcblk0p3): re-mounted 7886976b-c57d-43b2-aa41-04860776fe30 ro. Quota mode: disabled.
[...]

boot.cmd

[…]
env set rootfsargs_set ‘env set rootfsargs root=${mender_kernel_root} ro rootwait’
[…]

/etc/fstab

/etc/fstab:

/dev/root            /                    ext4       ro,noload                          0      0
proc                 /proc                proc       defaults                           0      0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0      0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0      0
tmpfs                /var/volatile        tmpfs      defaults                           0      0

# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0

Thanks and best regards,
Markus

2

Hi @MarkusN,

Thanks for reaching out! Concerning the re-mounting, good question. The logs are redacted, so there might be something hidden in those sections. My gut feeling is that there is some systemd unit which, without being noticed so far, re-mounts / to writable for whatever reason, and as part of its clean up stage then re-mounts to ro again. So the first place to go hunting would be all .mount units, I guess most is hidden there. Just like the data partitions 5 and 6 are also not visible in the fstab.

Right off the bat I don’t know if there’s a nice way to isolate such issues in the logs, so a very simplistic try might be building something like core-image-minimal or core-image-full-cmdline and seeing if the remounts still happen, then figuring out which package on top introduces the behavior.

Greetz,
Josef

3

Hello @TheYoctoJester !

Thank you for your reply, but there was nothing important missing from the logs (no error messages). We use SysV as our init system, and there is nothing in the runlevel scripts with mount or anything similar. Only GPIO, CAN, application start and so on.

The idea of trying a new core-image-minimal was also my next intention. I will give it a try.

And the thing with /etc/fstab was my mistake. It was a copy and paste error. Here is the entire file content:

# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    ext4       ro  0  0
proc                 /proc                proc       defaults  0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5  0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime  0  0
tmpfs                /var/volatile        tmpfs      defaults  0  0

# Where the U-Boot environment resides; for devices with SD card support ONLY!
/dev/mmcblk0p1       /uboot               auto       defaults,sync         0  2
/dev/mmcblk0p5       /data                ext4       sync,noatime,data=journal 0  2
/dev/mmcblk0p6       /mnt/config          ext4       ro,sync,noatime       0  0

Greetings,
Markus