HDMI stuck on rainbow screen but artifact boots

I have a Debian Stretch mender image, created with mender-convert 2.5.0. Its on a pi 3B, and the primary partition boots correctly and console output is displayed via HDMI. I then successfully installed a Debian Buster artifact to the secondary partition via standalone mender client.

The problem is that when the secondary partition boots, there is no HDMI output…just the rainbow screen. However the partition does completely boot up, as I can access it via SSH. In the hope of enabling HDMI output, I have been experimenting with the HDMI related settings in /uboot/config.txt. Unfortunately, none of those enable the HDMI.

For comparison, I mender converted a clean Buster image. So in this case Buster is running from the primary partition and there is nothing on the secondary. This image boots up and successfully outputs to HDMI. I then compared startup messages in the kernel logs between the two images, and thought I had found a smoking gun. The Buster image that correctly outputs to HDMI had the following in its kernel command line:

snd_bcm2835.enable_hdmi=1

This entry did not exist in the command line for the rainbow screen image. So I added it to cmdline.txt and rebooted. Unfortunately, it did not enable HDMI output.

What else is there to look for with this issue? The artifact installed on the secondary partition does not output to HDMI, but the full system image created with mender-convert will correctly show output on HDMI. Is there some other HDMI or console related setting in U-Boot to get past this?

For reference, here is the full kernel command line parameters for both cases.

HDMI output, running from primary partition:
Aug 5 17:31:35 raspberrypi kernel: [ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 fsck.repair=yes rootwait

No HDMI output, installed via artifact to secondary partition:
Jul 23 03:07:44 im3-rack1 kernel: [ 0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p3 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

I’ve done some more testing and while I don’t have a solution yet, I’ve found some interesting things and suspect this is a firmware related issue.

First test was to compare the kernel logs of the two buster images, looking for differences that would explain the lack of HDMI for the artifact version of Buster. And I did find firmware errors that did not exist in the other kernel log.

Jul 23 03:07:44 im3-rack1 kernel: [    0.000000] arch_timer: WARNING: Invalid trigger for IRQ194, assuming level low
Jul 23 03:07:44 im3-rack1 kernel: [    0.000000] arch_timer: WARNING: Please fix your firmware
Jul 23 03:07:44 im3-rack1 kernel: [    0.000000] arch_timer: WARNING: Invalid trigger for IRQ195, assuming level low
Jul 23 03:07:44 im3-rack1 kernel: [    0.000000] arch_timer: WARNING: Please fix your firmware

Followed later by:

Jul 23 03:07:44 im3-rack1 kernel: [    0.080271] raspberrypi-firmware soc:firmware: Request 0x00000002 returned status 0x80000001
Jul 23 03:07:44 im3-rack1 kernel: [    0.080330] raspberrypi-firmware soc:firmware: Attached to firmware from 2018-07-03T14:15:46, variant cmd unsupported
Jul 23 03:07:44 im3-rack1 kernel: [    0.090286] raspberrypi-firmware soc:firmware: Request 0x00000003 returned status 0x80000001

Not sure what these messages relate to, but in the successful log the warnings are not there and the subsequent firmware call looks like this:

Aug  9 19:37:29 im3-rack1 kernel: [    0.070087] raspberrypi-firmware soc:firmware: Attached to firmware from 2018-07-03 14:15

Next, I queried the HDMI status by using the tvservice application. So, when HDMI is displaying properly when booted from the primary partition, the command tvservice -s returns:

state 0x12000a [HDMI CEA (16) RGB lim 16:9], 1920x1080 @ 60.00Hz, progressive

But when HDMI is not displaying, the same command returns nothing and hangs.

This got me suspicious about the HDMI device being supported, and in the README for /uboot/overlays I saw an HDMI driver vc4-kms-v3d. So I added that to /uboot/config.txt and much to my surprise I got output via HDMI on the monitor!!! But the console scroll eventually stops and never reaches the login prompt. I think it stops when the firmware error occurs…those warnings and errors are still present in the kernel log, so understanding what they mean is the next step.

Can anyone offer insight on what those firmware errors refer to? This issue is only occurring when booting from the artifact on the secondary partition…and the Debian version on the secondary is different than the version on the primary. Booting from either partition works without issue if the Debian version is the same on both.

Maybe something in the uboot partition gets set at first boot, and when a different Debian version is run from the secondary there is an incompatibility affecting HDMI?

Thanks.