RaspiOS versions vs mender-convert versions

General Discussions
,
1

Apologies for the short post - I can follow up with debug info in due course (as necessary).

I recently started trying out mender (hosted.mender.io) and initially had success converting bootable images (and deployable mender rootfs artifacts) using:

  • mender-convert at version 2.3.0
  • RaspiOS Lite 2020-05-27

I have not been able to use any more recent versions of RaspiOS Lite, because the resulting ‘.img’ does not boot. I’ve been watching MEN-3941 and MEN-3945 with interest. The most recent version I’ve tried to use (with no success) is:

  • mender-convert on branch 2.3.x at commit dab3afdf
  • RaspiOS Lite 2021-01-11

I have two initial questions:

  • Q. Does this apparent failure in the integration tests for current HEAD of 2.3.x significant?
  • Q. How should I use mender-convert to convert a RaspiOS Lite 2021-01-11 system image? (assuming MEN-3945 is fixed)

Thanks!

2

Hi @wombat,

I was just able to run a converted image using the commit and RaspiOS Lite that you referenced. It took me 2 tries because it was a bit too big for my SDCard and I didn’t notice the dd failure but otherwise I can’t see anything wrong. I’m using a Raspberry Pi 3A device but I don’t think that should make a difference.

Can you provide a serial console log of your device with this configuration while it is booting?

Drew

3

Thanks for the quick reply Drew!

Apologies - forgot to mention that I’m running on RPi4.

Also, is this integration test failure at commit dab3afdf significant?

As far as I understand, I’ll need to get it booted normally then use rpi-eeprom-config to enable BOOT_UART to get early console; I have an old bus-pirate here (with its 3.3V UART mode) ready to connect…I’ll post when I have some logs.

Many thanks again!

4

I doubt if that test failure is significant but @oleorhagen should be able to confirm.

You should not need to jump through too many hoops to get a serial log. I just use one of these adapters and add ‘enable_uart=1’ to the /boot/config.txt file.

5

Nope, the test failure is not significant.

I have manually tested 2021-01-11 on RPi4 and it worked just fine.

Why is MEN-3945 critical to you @wombat ?

6

I’m getting a very strange behaviour:

  • When the UART is not enabled (like default config.txt), then the system does NOT boot. *Specifically, my config.txthasenable_uart=1` commented out.*
  • When the UART is enabled (enable_uart=1 in config.txt file), then the system boots OK.

Before running docker-mender-convert again, I pruned all my docker image layers/resources and then reran ‘docker-build’ (to build the mender-convert container image) at the current HEAD of branch 2.3.x.

I ran ‘docker-mender-convert’ this against RaspiOS Lite 2021-01-11.

With the UART enabled, I was able to boot the system. I then used rpi-eeprom-config to set BOOT_UART=1 in the EEPROM configuration, so that I could try and capture some console output to try and see what was happening (see below):

I’ve been using my old bus-pirate (with its 3.3V UART) to attach to the primary UART (GPIO 14/15) - captured logs are below.

Default non-booting (enable_uart=1 commented out):

PM_RSTS: 0x00001000
RPi: BOOTLOADER release VERSION:c305221a DATE: Sep  3 2020 TIME: 13:11:46 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1599135103 0xdf179f44 0x00d03114
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200
VLI: HUB2: 0xfff00000 0x24e6 MCU: 0xfff20000 0x15218
VL805 0xfff00000 0xfff20000
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
CID: 00275048534431364730014148f60122
CSD: 400e00325b590000737f7f800a400000
SD: bus-width: 4 spec: 2 SCR: 0x02358002 0x01000000
SD HOST: 250000000 CTL0: 0x00000f04 BUS: 50000000 Hz actual: 41666666 HZ div: 6 (3) status: 0x1fff0000 delay: 2
MBR: 0x00006000,  524288 type: 0x0c
MBR: 0x00086000,14327808 type: 0x83
MBR: 0x00e30000,14327808 type: 0x83
MBR: 0x01bda000, 1048576 type: 0x83
lba: 24576 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0
PM_RSTS: 0x00001000
Partition: 0
lba: 24576 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0
Read config.txt bytes     1800 hnd 0x00005ef5 hash '6910d15434027841'
recover4.elf not found (6)
recovery.elf not found (6)
Read start4.elf bytes  2215776 hnd 0x0000d407 hash '906e34e8700956d7'
Read fixup4.dat bytes     5429 hnd 0x000003f8 hash '1d0725a9904ebc93'
0x00d03114 0x00000000 0x0000003f
MEM GPU: 76 ARM: 948 TOTAL: 1024
Starting start4.elf @ 0xfec00200 partition 0
<NO FURTHER OUTPUT>

Booting OK (enable_uart=1 enabled):

PM_RSTS: 0x00001000
RPi: BOOTLOADER release VERSION:c305221a DATE: Sep  3 2020 TIME: 13:11:46 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1599135103 0xdf179f44 0x00d03114
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 3200
VLI: HUB2: 0xfff00000 0x24e6 MCU: 0xfff20000 0x15218
VL805 0xfff00000 0xfff20000
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
CID: 00275048534431364730014148f60122
CSD: 400e00325b590000737f7f800a400000
SD: bus-width: 4 spec: 2 SCR: 0x02358002 0x01000000
SD HOST: 250000000 CTL0: 0x00000f04 BUS: 50000000 Hz actual: 41666666 HZ div: 6 (3) status: 0x1fff0000 delay: 2
MBR: 0x00006000,  524288 type: 0x0c
MBR: 0x00086000,14327808 type: 0x83
MBR: 0x00e30000,14327808 type: 0x83
MBR: 0x01bda000, 1048576 type: 0x83
lba: 24576 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0
PM_RSTS: 0x00001000
Partition: 0
lba: 24576 oem: 'mkfs.fat' volume: ' boot       '
rsc 32 fat-sectors 4033 c-count 516190 c-size 1 r-dir 2 r-sec 0
Read config.txt bytes     1799 hnd 0x00005eed hash '68ca07b419dcf6ab'
recover4.elf not found (6)
recovery.elf not found (6)
Read start4.elf bytes  2215776 hnd 0x0000d407 hash '906e34e8700956d7'
Read fixup4.dat bytes     5429 hnd 0x000003f8 hash '1d0725a9904ebc93'
0x00d03114 0x00000000 0x0000003f
MEM GPU: 76 ARM: 948 TOTAL: 1024
Starting start4.elf @ 0xfec00200 partition 0


U-Boot 2020.01-g83cf4883ec (Mar 09 2021 - 08:12:31 +0000)

DRAM:  3.9 GiB
RPI 4 Model B (0xd03114)
MMC:   mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
568 bytes read in 49 ms (10.7 KiB/s)
## Executing script at 02400000
switch to partitions #0, OK
mmc0 is current device
...

Also, I tried setting config.txt options start_file=start4db.elf and fixup_file=fixup4db.dat, but this doesn’t yield anything different in the console output.

If I understand the RPi4 UART configuration correctly:

  • There are 6 UARTs (5 x PL011, 1 x ‘mini UART’), and the “primary UART” is connected to GPIO 14/15.
  • The “primary UART” is of type ‘mini UART’, which is disabled by default.
  • The ‘mini UART’ type requires a fixed VPU core clock frequency, and setting enable_uart=1 sets the GPU core clock to 250 MHz.

Could this (the GPU variable clock) be a factor?

A colleague of mine also sees this behaviour (with the same mender-converted image) on his RPi4 - i.e. that setting enable_uart=1 allows the system to boot OK. So for now we can continue developing, but it would be good to know what’s causing this behaviour.

Thanks again for all your help!

7

Hi!

All I am currently interested in, is making a bootable mender-converted system based on RaspiOS Lite (preferable a recent release).

Background - we are Linux users/developers but are not embedded Linux developers (yet) :-). In other words, the convenience of Debian and mender-convert is appreciated/valued as quick way for us to get up and running with OTA update prototyping; in the future we may look to Yocto or other routes to building a custom Linux for our application.

Initially, I got started very quickly with mender-convert (at tag 2.3.0) and RaspiOS Lite 2020-05-27 and everything was working as expected “out of the box”. However, I noticed that if I tried later versions of RaspiOS, the resulting img (from mender-convert) would not boot. Also, I had seen MEN-3941 and then later MEN-4395, so I was hoping/expecting to be able to use RaspiOS Lite 2020-01-11 to continue our evaluation of hosted.mender.io (v2.6) and mender-convert.

So, any guidance you can offer would be most appreciated; right now it looks like we either have to stick with the older 2020-05-27 RaspiOS Lite, or (assuming there are no other issues) just set enable_uart=1 in our config.txt in order to make our mender-converted 2021-01-11 RaspiOS Lite bootable.

Apologies if I’ve misunderstood anything or missed some documentation; this is all part of our exploration of mender. Thanks again to you and the mender team for a great set of tools!

Kind regards

Tom (wombat)

1 Like
8

Hi Tom,

It sounds like leaving the UART enabled is a reasonable workaround for you here. I suppose it is possible that the GPU clock could be part of the problem but it’s hard to say for certain.

Can you confirm that you do not see this behavior in the standard RaspiOS versions? The one big change that I think could affect this is the use of U-Boot with Mender.

When you get a board in the non-booting state, do you have a display of some kind connected? Is there any output?

Drew

9

Good instincts @drewmoseley I found this on the mailing list. So indeed it seems the GPU clock is divided for the UART, and this is causing some problems.

Indeed, I see that this blogpost mentions:

The enable_uart setting is required because U-Boot assumes the VideoCore firmware is configured to use the mini UART (rather than PL011) for the serial console. Without this, U-Boot will not boot at all.
10

Thanks for confirming (finding that uboot post)! I happened across this behaviour by chance, enabling the mini UART in order to get more dianostics and seeing that it actually allowed uboot to proceed. :slight_smile:

Regarding Drew’s question, yes I have an HDMI display attached. When the system is not booting (i.e. enable_uart=1 is not set present and console output stops at Starting start4.elf...), the HDMI output is black. The EEPROM HDMI debug screen has disappeared at this point in the boot process.

Tom

1 Like
11

Hi Tom,

Given the comment from @oleorhagen and the posts he references, I guess the behavior you describe with HDMI is unsurprising. It sounds like enable_uart=1 should be considered mandatory here and not just as a workaround.

Drew

12

Agreed. :slight_smile:

Given the enable_uart requirement of uboot in RPi4 has been commented in the uboot source for the last year, I’m a little curious to know how mender-convert 2.3.x was tested without noticing this issue before. Was it perhaps that the test systems had this option enabled already so it went unnoticed?

13

Yeah, my guess is all testing has been done on devices with UART enabled. I know that’s the first thing I do on any of these devices.

@oleorhagen would it make sense to modify mender-convert to ensure that the UART is enabled?

Drew

14

Yes, it is tested with UART enabled by default (just gives more output from which you can debug).

It never occured to me that this might come back and bite us like this, hah.

We are having an internal discussion about how to approach this right now, I will get back to you within the day with a response on what we will decide to do here :slight_smile:

15

We decided to enable uart for all boards. See pull request here.

1 Like
16

Thanks again to everyone on the Mender team for the quick follow-up and responses to this topic! It’s much appreciated!

1 Like