Does mender change u-boot dts?

Have you seen the " Problem using ‘dtoverlay=pi3-disable-bt’" section in “Known Issues” @ Raspberry Pi Compute Module 3?

Also it is the boot firmware that chooses which device tree to load, and the boot scripts in meta-mender re-use that.

what is the actual device tree which is used by U-Boot in this case

This question can lead to confusion, U-boot has a built-in (part of its source code) device tree which it is used so none of the mentioned dtb files are used by U-boot.

The dtb files that you are referring to are something that is passed along to the Linux kernel and parsed by Linux kernel only.