The OpenGL renderer in VMware Workstation 16.2.x+ no longer works with VirtualGL

[DRC]

VMware Workstation v15.5 and later uses a GPU sandbox process that translates OpenGL commands from the guest into either Vulkan or OpenGL commands and forwards them to the GPU in the host. Starting with VMware Workstation 16.2.0 (https://docs.vmware.com/en/VMware-Workstation-Pro/16.2.0/rn/VMware-Workstation-1620-Pro-Release-Notes.html), the GPU sandbox uses a Vulkan renderer by default on Linux hosts with supported GPUs, so VirtualGL no longer has any effect. nVidia's Vulkan implementation doesn't require an X display in order to access the GPU. If it detects that it is running in an X proxy, it does something VirtualGL-like (i.e. it uses a swap chain that reads back rendered frames from the GPU and draws them into the X window using XPutImage().) Otherwise, it uses a no-op swap chain. Thus, if using nVidia's Vulkan implementation on a new enough nVidia GPU, VMware Workstation doesn't require VirtualGL in order to use the GPU, even if VMware is running headless or in an X proxy. However, you can also set

mks.enableVulkanRenderer = "FALSE"
mks.enableGLRenderer = "TRUE"

in the .vmx file or in /etc/vmware/config. When you do that, the GPU sandbox uses a "legacy" OpenGL renderer and accesses the GPU through the X server specified in DISPLAY, as it used to do prior to VMware Workstation v16.2.x. Unfortunately, VMware's OpenGL renderer no longer works properly with VirtualGL, for unknown reasons. Looking at mksSandbox.log, the sandbox process appears to be using the unaccelerated OpenGL implementation in the X proxy. I can think of two explanations:

1. Because mksSandbox is a separate process, LD_PRELOAD doesn't get passed down to it from vglrun. I have tried renaming mksSandbox to _mksSandbox and creating a script called mksSandbox that invokes _mksSandbox using vglrun. However, that doesn't work.
2. The sandbox may be using XCB-GLX, in which case it's probably a lost cause. We interpose only the bare minimum of XCB-GLX necessary to make Qt5 work. Interposing the entire XCB-GLX API would only be possible with the GLX back end, and there are so many XCB-GLX functions that a full XCB-GLX interposer would be tedious to implement and difficult to test and maintain.

Closing as "won't implement" for now, since VMware's OpenGL renderer is a legacy feature. But if someone knows something I don't, please comment below.