RPi will not initialize with screen connected. Works if hot plugged after starting without it.

[mrkr4g]

Raspberry Pi 3b+
TiltPi image running on Buster
RPi 3.5" GPIO Touchscreen
3.0A power supply

So I got the driver installed and the pi will not start up from a cold start. When it rebooted after driver install, all was fine. It starts up and displays on the LCD as expected. But if I do a shutdown and then try to start it up it does nothing. Green light never even blinks.
If I pull the screen off, even for just a second, the initialization and boot sequences go fine and it will start up and run. Then again, it will do a reboot but not a shutdown and restart sequence.

Because of this I can not go ahead and close up the case since I have to pull the screen off to start up.

Advice has been to check in initialization scripts in /etc/rc.local and /boot/config.txt but it never even gets to these as the card is never read in the first place.

Edit: I should also mention this behavior is the same on other system images, including bullseye and the latest bookworm from the official site. I am just using the TiltPi Buster image because I have had no luck getting TiltPi to install from repos on any of the stock Pi images so I went with that one to at least get Tilt working.