Good instincts @drewmoseley I found this on the mailing list. So indeed it seems the GPU clock is divided for the UART, and this is causing some problems.
Indeed, I see that this blogpost mentions:
The enable_uart setting is required because U-Boot assumes the VideoCore firmware is configured to use the mini UART (rather than PL011) for the serial console. Without this, U-Boot will not boot at all.