Mac address seems to be getting hardcoded onto the SD card

General Discussions
#1

This is a curious one and it may not be Mender related, but here goes anyways. I’m on Raspberry Pi 3B’s / 3B+'s and what I’m observing is that upon booting a fresh image from Yocto on the Pi it seems to be somehow locking the mac address of that Pi’s Ethernet adapter into the SD card. If I move that SD card to another Raspberry Pi, it will boot up and be using the mac address from the first Pi. I’m seeing this both in Linux and U-Boot, though I can’t find any files that are changing in the boot partition.

Has anyone seen anything like this before?

Edit: After further investigation, it looks like the address gets locked in after first booting into Linux, and a number of u-boot variables appear in printenv after that first Linux boot, including a mender variable. So I’m pretty sure it’s something Mender. Looking through the doc now. (I’m still on 1.7)

#2

Culprit seems to be here: https://github.com/mendersoftware/uboot-mender/blob/07a59f430cb0428dcf525caf9394172a1290ef1b/board/raspberrypi/rpi/rpi.c#L305

#3

Interesting.

By default the RPi will query the boot firmware for the MAC address and this is how it gets a “unique” address for each device.

Interestingly though as you already might have figured out, the code that you link to will prioritize the ethaddr environment variable over what the response is from boot firmware.

Meaning that if you run e.g:

# setenv ethaddr 00:11:22:33:44:55
# saveenv

This address will be used and this is stored on the SD card, meaning that it will follow when you move the SD card to another device.

This might also happen in mender_setup where do call saveenv meaning that if ethaddr is set to something when mender_setup is run it will save it to the SD card.

#4

Yeah, I’m not sure what the best general solution will be. For my purposes I can just change that line so it always sets the Mac address from the usbethaddr variable, but I believe the ethaddr flag is meant to be modified by the user.

#5

Also, to be clear, the actual issue is that Linux is using the mac address from ethaddr. So hopefully changing that line in u-boot would fix it in Linux as well, but I’m not aware of the mechanism by which Linux gets the address. I can find the hard-coded (not actual device) MAC address within the /proc/device-tree but I’m not 100% sure that that’s where the ethernet driver gets the MAC from.

#6

It should be visible in /proc/cmdline and that is how kernel picks it up.

#7

I’d checked :slight_smile: Didn’t see anything that seemed relevant to that.

root@raspberrypi3:~# cat /proc/cmdline
8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200  rootfstype=ext4 rootwait     root=/dev/mmcblk0p2
#8

@mirzak So the problem mostly stems from the original u-boot source that I linked above but the saveenv seems to be the thing that causes the damage. I’m also seeing that a card that I boot in a RPi 3B fails to boot in a 3B+ and I suspect that some other environment variable that’s being saved is responsible for it. Is there an alternative to saveenv? Can we just save specific variables to permanent storage?

Right now I’m working on a patch for u-boot for the Mac address thing, but I’m worried about what other side effects happen due to doing the saveenv. Right now we’re working on finding and removing all Raspberry Pi 3B’s from the office so we don’t have the cross-over failure.

#9

What about running “env default -a” before the save?

When I ran that, these variables were changed:

--- <unnamed>
+++ <unnamed>
@@ -1,13 +1,9 @@
-U-Boot> printenv
+U-Boot> printenv      
 altbootcmd=run mender_altbootcmd; run bootcmd
 arch=arm
 baudrate=115200
 board=rpi
-board_name=3 Model B+
-board_rev=0xD
-board_rev_scheme=1
-board_revision=0xA020D3
+board_name=rpi
 boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
 boot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootarm.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroi
 boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf
@@ -16,32 +12,20 @@
 boot_script_dhcp=boot.scr.uimg
 boot_scripts=boot.scr.uimg boot.scr
 boot_targets=mmc0 usb0 pxe dhcp 
-bootargs=8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200  rootfstype=ext4 rootwait     root=${mender_kernel_root}
 bootcmd=run distro_bootcmd
 bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bo;
 bootcmd_mmc0=setenv devnum 0; run mmc_boot
 bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi
 bootcmd_usb0=setenv devnum 0; run usb_boot
-bootcount=1
+bootcount=0
 bootdelay=2
-bootfstype=fat
 bootlimit=1
 cpu=armv7
-devnum=0
-devplist=1
-devtype=mmc
 dhcpuboot=usb start; dhcp u-boot.uimg; bootm
 distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
 efi_dtb_prefixes=/ /dtb/ /dtb/current/
-ethaddr=b8:27:eb:a0:aa:05
-fdt_addr=2eff9600
 fdt_addr_r=0x02600000
 fdt_high=ffffffff
-fdtaddr=2eff9600
-fdtcontroladdr=3b3ae9b0
-fdtfile=bcm2837-rpi-3-b-plus.dtb
-fileaddr=2400000
-filesize=12c
 initrd_high=ffffffff
 kernel_addr_r=0x00080000
 load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
@@ -53,7 +37,6 @@
 mender_check_saveenv_canary=1
 mender_dtb_name=bcm2710-rpi-cm3.dtb
 mender_kernel_name=uImage
-mender_saveenv_canary=1
 mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_nai
 mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi
 mender_uboot_boot=mmc 0:1
@@ -69,14 +52,13 @@
 scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
 scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
 scriptaddr=0x02400000
-serial#=0000000038a0aa05
 soc=bcm283x
 stderr=serial,vidconsole
 stdin=serial,usbkbd
 stdout=serial,vidconsole
 upgrade_available=0
 usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi
-usbethaddr=b8:27:eb:a0:aa:05
 vendor=raspberrypi
 
-Environment size: 5960/16379 bytes
+Environment size: 5309/16379 bytes
+U-Boot>

Edit: @mirzak: highlight as “patch”

#10

First of all thanks for digging in to this and reporting back your findings. The findings are relevant and actually also explains some things that I have experienced as well but never understood (why moving SD card from RPi3 Model B to B+ did not work)

I am not aware of any alternatives to saveenv and you always write the whole environment in one go, this is also something that makes changes to the environment atomic and this is important to keep intact.

Right now I’m working on a patch for u-boot for the Mac address thing, but I’m worried about what other side effects happen due to doing the saveenv. Right now we’re working on finding and removing all Raspberry Pi 3B’s from the office so we don’t have the cross-over failure.

The purpose of the saveenv command is to detect inconsistencies in environment configuration between U-boot and the user-space. This is what mender_saveenv_canary=1 is used for and this is the purpose of the saveenv. The Mender client will actually check the presence of this variable and complain if it is not there. That is at least my understanding as I have not looked in to these specific parts in a while.

Note that the saveenv is only executed once, during the first boot.

Trying to run env default -a before calling saveenv will force it to re-write the U-boot environment on each boot, this is not desirable as it will cause “wear” on the SD card and increase chances of breaking blocks. Something to avoid to increase longevity of your device.

With all that said, I do not really see any major side-effects/downsides of this behaviour expect that you might not be able to move an SD card that has booted once from a RPi3 B+ to RPi 3 B. But I would argue that every time you move the SD card to between devices there are other reasons on why you would re-provision it again to get a fresh start, even if it is from RPi3 B+ to RPi3 B+.

One reason is the device identity which is used by Mender and this is typically a hardware bound identifier. Now in this specific case it is using the defaults (MAC address, which was carried over due to the U-boot environment) but this is something that is configurable and might as well have been something else that is not related to the U-boot environment.

And would not be to worried about the saveenv causing other problems as this is the environment that your device boots with and the result of the saveenv will only save it to disk, but it would otherwise contain these variables even if we did not do the saveenv.

#11

Glad that I helped, and thanks for the detailed response!

I think I mostly agree. This is a problem that’s only occurring for us because we’re in a rapid build-test-deploy cycle where we’re testing on one board before moving the SD card to another board. Some of the other research I’ve done since my last post implies that u-boot was never intended to be moved from one board to another, especially with respect to the ethaddr variable.

#12

@mirzak Is there a way I can fix an image afterwards and change the MAC address that it hardcoded?

What file can I update? Is it possible?

I tried mender bootstrap but that didn’t work.