Unable to perform the OTA update on BeagleBone Green eMMC

General Discussions
21

@kacf Thank you for reply.

  1. I have no any screen by hand, so uart and ssh is everything I have.
  2. I believe the bootloaders made their work correctly, so the problem is in configuration.
    I managed to successfully load the SD card image, not eMMC and I start to realize where the problem resides.

Here’s the contents of grub.cfg from eMMC flasher image (that’s failing to boot):

# Start of ---------- 00_mender_grubenv_defines_grub.cfg ----------
mender_rootfsa_part=2
mender_rootfsb_part=3
mender_grub_storage_device=hd0
kernel_imagetype=kernel
initrd_imagetype=initrd
mender_kernel_root_base=/dev/mmcblk1p
# End of ---------- 00_mender_grubenv_defines_grub.cfg ----------
# Start of ---------- 01_console_bootargs_grub.cfg ----------
set console_bootargs="console=tty0,115200n8 console=ttyS0,115200n8 console=ttyO0,115200n8 console=ttyAMA0,115200n8"
# End of ---------- 01_console_bootargs_grub.cfg ----------
# Start of ---------- 02_root_bootargs_grub.cfg ----------
set rootargs="rootwait"
# End of ---------- 02_root_bootargs_grub.cfg ----------
# Start of ---------- 05_mender_setup_grub.cfg ----------
# See the fw_printenv script for how this works.

function maybe_pause {
    # By default we do nothing. debug-pause PACKAGECONFIG replaces this so we
    # can pause at strategic places.
    echo
}

drop_to_grub_prompt="no"
function maybe_drop_to_grub_prompt {
    # By default we do nothing. force-grub-prompt PACKAGECONFIG replaces this so we
    # can bypass boot and stop at the grub prompt.
    echo
}

# Load environment.

MENDER_ENV1=${prefix}/mender_grubenv1/env
MENDER_ENVPREFIX1=${prefix}/mender_grubenv1/
MENDER_LOCK1=${prefix}/mender_grubenv1/lock
MENDER_LOCKSUM1=${prefix}/mender_grubenv1/lock.sha256sum
MENDER_ENV2=${prefix}/mender_grubenv2/env
MENDER_ENVPREFIX2=${prefix}/mender_grubenv2/
MENDER_LOCK2=${prefix}/mender_grubenv2/lock
MENDER_LOCKSUM2=${prefix}/mender_grubenv2/lock.sha256sum

function mender_check_and_restore_env {
    if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX2} --check ${MENDER_LOCKSUM2}; then
        load_env --skip-sig --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        save_env --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        editing=0
        save_env --file ${MENDER_LOCK2} editing
        if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX2} --check ${MENDER_LOCKSUM2}; then
            echo "Environment 2 still corrupt after attempted restore. Halting."
            halt
        fi
    elif ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX1} --check ${MENDER_LOCKSUM1}; then
        load_env --skip-sig --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        save_env --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        editing=0
        save_env --file ${MENDER_LOCK1} editing
        if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX1} --check ${MENDER_LOCKSUM1}; then
            echo "Environment 1 still corrupt after attempted restore. Halting."
            halt
        fi
    fi
}

function mender_save_env {
    # Save redundant environment.
    editing=1
    save_env --file ${MENDER_LOCK2} editing
    save_env --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
    editing=0
    save_env --file ${MENDER_LOCK2} editing

    editing=1
    save_env --file ${MENDER_LOCK1} editing
    save_env --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
    editing=0
    save_env --file ${MENDER_LOCK1} editing
}

function mender_check_grubenv_valid {
    if [ "${mender_boot_part}" != "${mender_rootfsa_part}" -a "${mender_boot_part}" != "${mender_rootfsb_part}" ]; then
        return 1
    fi

    if [ "${bootcount}" != "0" -a "${bootcount}" != "1" ]; then
        return 1
    fi

    if [ "${upgrade_available}" != "0" -a "${upgrade_available}" != "1" ]; then
        return 1
    fi

    return 0
}

mender_check_and_restore_env

# Now load environment.

# Skipping signatures?? Yes, because these values will change over time, so they
# cannot be signed. There is also no checksum facility that will work for
# changing values. Instead we check their content for validity.
load_env --skip-sig --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id

if ! mender_check_grubenv_valid; then
    if [ "${check_signatures}" == "enforce" ]; then
        echo "Unverified environment and signatures enabled. Halting."
        halt
    fi
fi

if [ "${upgrade_available}" = "1" ]; then
    if [ "${bootcount}" != "0" ]; then
        echo "Rolling back..."
        if [ "${mender_boot_part}" = "${mender_rootfsa_part}" ]; then
            mender_boot_part="${mender_rootfsb_part}"
        else
            mender_boot_part="${mender_rootfsa_part}"
        fi
        upgrade_available=0
        bootcount=0
    else
        echo "Booting new update..."
        bootcount=1
    fi

    mender_save_env
fi

regexp (.*),(.*) $root -s mender_grub_storage_device
# End of ---------- 05_mender_setup_grub.cfg ----------
# Start of ---------- 06_systemd_machine_id_grub.cfg ----------
if test -n "${mender_systemd_machine_id}"; then
   systemd_bootargs="systemd.machine_id=${mender_systemd_machine_id}"
fi
# End of ---------- 06_systemd_machine_id_grub.cfg ----------
# Start of ---------- 10_bootargs_grub.cfg ----------
set bootargs="${bootargs} ${console_bootargs} ${rootargs} ${systemd_bootargs}"
# End of ---------- 10_bootargs_grub.cfg ----------
# Start of ---------- 90_mender_boot_grub.cfg ----------
maybe_drop_to_grub_prompt

if test -e (${mender_grub_storage_device},gpt${mender_boot_part})/; then
    ptable_type=gpt
else
    ptable_type=msdos
fi

if test -n "${mender_rootfsa_uuid}" -a test -n  "${mender_rootfsb_uuid}"; then
    if [ "${mender_boot_part}" = "${mender_rootfsa_part}" ]; then
        mender_root="PARTUUID=${mender_rootfsa_uuid}"
    elif [ "${mender_boot_part}" = "${mender_rootfsb_part}" ]; then
        mender_root="PARTUUID=${mender_rootfsb_uuid}"
    fi
else
    mender_root="${mender_kernel_root_base}${mender_boot_part}"
fi

if [ "${drop_to_grub_prompt}" = "no" ]; then
    if linux (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${kernel_imagetype} root=${mender_root} ${bootargs}; then
        if test -n "${initrd_imagetype}" -a test -e (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${initrd_imagetype}; then
            initrd (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${initrd_imagetype}
        fi
        maybe_pause "Pausing before booting."
        boot
    fi
    maybe_pause "Pausing after failed boot."
fi
# End of ---------- 90_mender_boot_grub.cfg ----------
# Start of ---------- 95_mender_try_to_recover_grub.cfg ----------
if [ "${drop_to_grub_prompt}" = "no" ]; then
    if [ "${upgrade_available}" == "1" ]; then
        reboot
    fi
fi
# End of ---------- 95_mender_try_to_recover_grub.cfg ----------
# Start of ---------- 99_mender_end_of_grub.cfg ----------
if [ "${drop_to_grub_prompt}" = "yes" ]; then
    echo "Dropping to grub prompt intentionally."
    sleep --interruptible 10 --verbose
else
    echo "Dropping to grub prompt for unknown reason. Should never get here."
    sleep --interruptible 10 --verbose
fi
menuentry 'Dummy Entry for Debug.' 'Wait' {
   echo Menu Entry for debug/command prompt access
}
# End of ---------- 99_mender_end_of_grub.cfg ----------

And here’s the same file for the generic SD card image:

# Start of ---------- 00_mender_grubenv_defines_grub.cfg ----------
mender_rootfsa_part=2
mender_rootfsb_part=3
mender_grub_storage_device=hd0
kernel_imagetype=kernel
initrd_imagetype=initrd
mender_kernel_root_base=/dev/mmcblk0p
# End of ---------- 00_mender_grubenv_defines_grub.cfg ----------
# Start of ---------- 01_console_bootargs_grub.cfg ----------
set console_bootargs="console=tty0,115200n8 console=ttyS0,115200n8 console=ttyO0,115200n8 console=ttyAMA0,115200n8"
# End of ---------- 01_console_bootargs_grub.cfg ----------
# Start of ---------- 02_root_bootargs_grub.cfg ----------
set rootargs="rootwait"
# End of ---------- 02_root_bootargs_grub.cfg ----------
# Start of ---------- 05_mender_setup_grub.cfg ----------
# See the fw_printenv script for how this works.

function maybe_pause {
    # By default we do nothing. debug-pause PACKAGECONFIG replaces this so we
    # can pause at strategic places.
    echo
}

drop_to_grub_prompt="no"
function maybe_drop_to_grub_prompt {
    # By default we do nothing. force-grub-prompt PACKAGECONFIG replaces this so we
    # can bypass boot and stop at the grub prompt.
    echo
}

# Load environment.

MENDER_ENV1=${prefix}/mender_grubenv1/env
MENDER_ENVPREFIX1=${prefix}/mender_grubenv1/
MENDER_LOCK1=${prefix}/mender_grubenv1/lock
MENDER_LOCKSUM1=${prefix}/mender_grubenv1/lock.sha256sum
MENDER_ENV2=${prefix}/mender_grubenv2/env
MENDER_ENVPREFIX2=${prefix}/mender_grubenv2/
MENDER_LOCK2=${prefix}/mender_grubenv2/lock
MENDER_LOCKSUM2=${prefix}/mender_grubenv2/lock.sha256sum

function mender_check_and_restore_env {
    if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX2} --check ${MENDER_LOCKSUM2}; then
        load_env --skip-sig --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        save_env --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        editing=0
        save_env --file ${MENDER_LOCK2} editing
        if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX2} --check ${MENDER_LOCKSUM2}; then
            echo "Environment 2 still corrupt after attempted restore. Halting."
            halt
        fi
    elif ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX1} --check ${MENDER_LOCKSUM1}; then
        load_env --skip-sig --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        save_env --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
        editing=0
        save_env --file ${MENDER_LOCK1} editing
        if ! hashsum --hash sha256 --prefix ${MENDER_ENVPREFIX1} --check ${MENDER_LOCKSUM1}; then
            echo "Environment 1 still corrupt after attempted restore. Halting."
            halt
        fi
    fi
}

function mender_save_env {
    # Save redundant environment.
    editing=1
    save_env --file ${MENDER_LOCK2} editing
    save_env --file ${MENDER_ENV2} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
    editing=0
    save_env --file ${MENDER_LOCK2} editing

    editing=1
    save_env --file ${MENDER_LOCK1} editing
    save_env --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id
    editing=0
    save_env --file ${MENDER_LOCK1} editing
}

function mender_check_grubenv_valid {
    if [ "${mender_boot_part}" != "${mender_rootfsa_part}" -a "${mender_boot_part}" != "${mender_rootfsb_part}" ]; then
        return 1
    fi

    if [ "${bootcount}" != "0" -a "${bootcount}" != "1" ]; then
        return 1
    fi

    if [ "${upgrade_available}" != "0" -a "${upgrade_available}" != "1" ]; then
        return 1
    fi

    return 0
}

mender_check_and_restore_env

# Now load environment.

# Skipping signatures?? Yes, because these values will change over time, so they
# cannot be signed. There is also no checksum facility that will work for
# changing values. Instead we check their content for validity.
load_env --skip-sig --file ${MENDER_ENV1} bootcount mender_boot_part upgrade_available mender_systemd_machine_id

if ! mender_check_grubenv_valid; then
    if [ "${check_signatures}" == "enforce" ]; then
        echo "Unverified environment and signatures enabled. Halting."
        halt
    fi
fi

if [ "${upgrade_available}" = "1" ]; then
    if [ "${bootcount}" != "0" ]; then
        echo "Rolling back..."
        if [ "${mender_boot_part}" = "${mender_rootfsa_part}" ]; then
            mender_boot_part="${mender_rootfsb_part}"
        else
            mender_boot_part="${mender_rootfsa_part}"
        fi
        upgrade_available=0
        bootcount=0
    else
        echo "Booting new update..."
        bootcount=1
    fi

    mender_save_env
fi

regexp (.*),(.*) $root -s mender_grub_storage_device
# End of ---------- 05_mender_setup_grub.cfg ----------
# Start of ---------- 06_systemd_machine_id_grub.cfg ----------
if test -n "${mender_systemd_machine_id}"; then
   systemd_bootargs="systemd.machine_id=${mender_systemd_machine_id}"
fi
# End of ---------- 06_systemd_machine_id_grub.cfg ----------
# Start of ---------- 10_bootargs_grub.cfg ----------
set bootargs="${bootargs} ${console_bootargs} ${rootargs} ${systemd_bootargs}"
# End of ---------- 10_bootargs_grub.cfg ----------
# Start of ---------- 90_mender_boot_grub.cfg ----------
maybe_drop_to_grub_prompt

if test -e (${mender_grub_storage_device},gpt${mender_boot_part})/; then
    ptable_type=gpt
else
    ptable_type=msdos
fi

if test -n "${mender_rootfsa_uuid}" -a test -n  "${mender_rootfsb_uuid}"; then
    if [ "${mender_boot_part}" = "${mender_rootfsa_part}" ]; then
        mender_root="PARTUUID=${mender_rootfsa_uuid}"
    elif [ "${mender_boot_part}" = "${mender_rootfsb_part}" ]; then
        mender_root="PARTUUID=${mender_rootfsb_uuid}"
    fi
else
    mender_root="${mender_kernel_root_base}${mender_boot_part}"
fi

if [ "${drop_to_grub_prompt}" = "no" ]; then
    if linux (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${kernel_imagetype} root=${mender_root} ${bootargs}; then
        if test -n "${initrd_imagetype}" -a test -e (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${initrd_imagetype}; then
            initrd (${mender_grub_storage_device},${ptable_type}${mender_boot_part})/boot/${initrd_imagetype}
        fi
        maybe_pause "Pausing before booting."
        boot
    fi
    maybe_pause "Pausing after failed boot."
fi
# End of ---------- 90_mender_boot_grub.cfg ----------
# Start of ---------- 95_mender_try_to_recover_grub.cfg ----------
if [ "${drop_to_grub_prompt}" = "no" ]; then
    if [ "${upgrade_available}" == "1" ]; then
        reboot
    fi
fi
# End of ---------- 95_mender_try_to_recover_grub.cfg ----------
# Start of ---------- 99_mender_end_of_grub.cfg ----------
if [ "${drop_to_grub_prompt}" = "yes" ]; then
    echo "Dropping to grub prompt intentionally."
    sleep --interruptible 10 --verbose
else
    echo "Dropping to grub prompt for unknown reason. Should never get here."
    sleep --interruptible 10 --verbose
fi
menuentry 'Dummy Entry for Debug.' 'Wait' {
   echo Menu Entry for debug/command prompt access
}
# End of ---------- 99_mender_end_of_grub.cfg ----------

f77e@hp:/opt/bc-deployment-tools/mender-convert$ diff grub.cfg.emmc grub.cfg.sdcard 
7c7
< mender_kernel_root_base=/dev/mmcblk1p
---
> mender_kernel_root_base=/dev/mmcblk0p
f77e@hp:/opt/bc-deployment-tools/mender-convert$

Currently, for experiment purposes I have manually changed this variable value to /dev/mmcblk0p and have succeeded to go further in a boot progress, but still it hangs on the current strings

[   37.537670] remoteproc remoteproc0: powering up wkup_m3
[   37.547441] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 217168
[   37.569707] remoteproc remoteproc0: remote processor wkup_m3 is now up
[   37.569731] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x193
[   37.638726] PM: bootloader does not support rtc-only!
Give root password for maintenance
(or press Control-D to continue):

Whenever if I’m using the "successful’ image, it shows me the login prompt.

22

I was succeeded to boot by setting MENDER_STORAGE_DEVICE_BASE="/dev/mmcblk0p"
in configs/beaglebone_black_debian_emmc_config`.
The eMMC flasher functions are triggered from the last line of uEnv.txt file (which I believe now is not working).

Is there’s a similar way for GRUB to provide the entrypoint of flashing process ?

23

Sorry for flooding, just reporting my progress as I go.

I found the way to pass the init= kernel parameter from grub.cfg.

But there’s another question.
In order to prepare the u-boot based flasher image using mender-convert, it’s needed (at least for me) just to provide the prepared rootfs containing changed uEnv.txt.

Is there’s a way to change bootloader config only by mender-convert instructions ?

24

It’s possible that this is related to this page. See the section on “bootloader does not support rtc-only” and the suggestions.

25

Kind of, but this is highly custom. Take a look at the Hooks/Overrides documentation for mender-convert.

uEnv.txt is not really used by the mender-convert integration, we just rely on U-Boot to call UEFI binaries and use that to do a normal boot using GRUB. All extra steps that U-Boot needs to carry out therefore require custom modifications.

26

It seems like it doesn’t affect the boot process, it was just a last message before logs stopped to happen. The real problem was that grub has passed a wrong root partition in a kernel command line args. In anyway, I very appreciate that you took care about it.

Thanks, I already triggered the flasher by substituting grub.cfg using platform_modify hook.

I have another question, hopefully the last one.
Looking at LED’s I see that flasher is running, but I see no logs, whenever I know it’s logging.

So I have manually edited grub.cfg and here’s its changed section:

# Start of ---------- 10_bootargs_grub.cfg ----------
#set bootargs="${bootargs} ${console_bootargs} ${rootargs} ${systemd_bootargs}"
set bootargs="init=/opt/scripts/tools/eMMC/init-eMMC-flasher-Mender.sh ${console_bootargs} ${rootargs} ${systemd_bootargs}"
# End of ---------- 10_bootargs_grub.cfg ----------

Having init substituted by another script, I see no logs.
Could you please direct me where to look at ?

27

I don’t know why that wouldn’t work, but on a general basis, I would suggest using an initrd to do the flashing instead. These are a bit more standardized, and I do believe logs would work inside them. Looking at the initramfs-tools man page might be a start, and the various hooks in /usr/share/initramfs-tools/ might be a place to add the flasher.

28

Finally, I’ve managed to perform the update.

I have used the systemd unit with nice=-20 to autostart the flashing procedure on boot. Got tried it a lot of times and it works flawlessly.

Things to consider for those few that might run BB on eMMC:

  • grub.cfg should point to a proper root partition
  • fstab in your target filesystem differs from one at the ‘flasher’ filesystem
  • /data/mender/mender.conf should point to a proper partitions as well

@kacf your help was invaluable. Great thanks !

1 Like
29

Glad to hear you worked it out!