Unable to use gaming mode with 4K HDR display - Solved

[GeddesJ]

Describe the bug

Hi. I managed to figure out how to solve this on my own, but thought I would submit a bug to document this in case anyone else has the same issue.
I installed Bazzite about a month ago and had no issues getting the gaming mode to work with the old 1080p TV I was using at the time. My system specs should be the poster child for getting gamescope to work correctly. (AMD CPU + GPU) :)
Recently, however, I upgraded to a fancy new 4k HDR display and I was greatly disappointed when the first time I booted the system with the display plugged in I only got a black screen, before it fell back into loading KDE.

What did you expect to happen?

I was hoping that gaming mode should "just work" with any display you plug in without needing any additional configuration.

Output of rpm-ostree status

State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:stable
                   Digest: sha256:25e6c62e647e7bef2a461f24d0c7fea72a86fb95edb66bae00f350e42364bf55
                  Version: nightly (2024-04-26T17:04:40Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf /etc/modprobe.d/deck-blacklist.conf" ' 

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-deck:stable
                   Digest: sha256:4e41469b61e296775d65ca85506142775ad7a927242619683de7e75b0317f233
                  Version: main (2024-04-26T01:18:26Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf /etc/modprobe.d/deck-blacklist.conf" '

Hardware

Operating System: Bazzite 40
KDE Plasma Version: 6.0.3
KDE Frameworks Version: 6.0.0
Qt Version: 6.6.2
Kernel Version: 6.8.7-302.fsync.fc40.x86_64 (64-bit)
Graphics Platform: Wayland
Processors: 12 × AMD Ryzen 5 7600 6-Core Processor
Memory: 14.8 GiB of RAM
Graphics Processor: AMD Radeon RX 7800 XT
Manufacturer: Micro-Star International Co., Ltd.
Product Name: MS-7E28
System Version: 2.0

Monitor: Philips 436M6VBR

Extra information or context

The first thing I tried was plugging my old TV back in to see if maybe coincidentally an update had broken game mode. It still worked with the TV, but what I found is that if I unplugged HDMI cable with game mode running and then plugged into the 4K display, it would work! And even detect the correct resolution and HDR. This then lead me down a path of figuring out how to configure gamescope.

Solution
Create a file ~/.config/environment.d/gamescope-session-plus.config
The first thing I tried was setting the contents to:
SCREEN_HEIGHT=2160 SCREEN_WIDTH=3840
This was enough to get the system booting into gaming mode once again, however often the colours were all messed up until tweaking the display settings. And it would frequently crash when trying to access the Steam menu.
Looking at the logs in ~/.local/share/Steam/logs/steamui_system.txt I saw this error a few times around when it would crash:
Error: CX11WindowProperty: failed to read window property: GAMESCOPE_COLOR_APP_WANTS_HDR_FEEDBACK
Note that I had HDR turned on in the UI, but disabling didn't make any difference to the instability.

With the understanding that my issue was potentially HDR related, I started looking into the gamescope command line arguments and found if I set my gamescope-session-plus.config to this:
GAMESCOPECMD="gamescope -e -f --hdr-enabled --hdr-itm-enable --adaptive-sync --xwayland-count 2"
Now my gaming mode works flawlessly without any crashes or funky colours. The flags to force HDR are what fixed it.

Hopefully this is helpful for anyone else who might have this issue with gaming mode.