Add support for Streamline or add EAT hooking

[thewhitegoatcb]

Description

Adding support for application that use https://github.com/NVIDIAGameWorks/Streamline/ would be nice.

I did a more in depth analysis of why the generic hooking solution does not work there, here's how the streamline wrapper works under the hood:

1. Application imports a wrapper call from sl.interposer.dll.
2. The wrapper loads the .dll via LoadLibraryW.
3. Manually walking the Export Directory to get the wrapped api, (without calling GetProcAddress).
4. Calling the API directly, skipping any IAT/GetProcAddress hooks RenderDoc has set.

Here's an example of the wrapper for D3D11CreateDeviceAndSwapChain/D3D11CreateDevice
https://github.com/NVIDIAGameWorks/Streamline/blob/7ac42e47c7dd55b5b6dd6176c0228048636541b2/source/core/sl.interposer/d3d11/d3d11.cpp#L43

Possible solution would be either to hook Streamline explicitly or add EAT hooks for more generic support.

Environment

• RenderDoc version: 1.31
• Operating System: Windows 10 x64
• Graphics API: D3D11/D3D12

[Baldur Karlsson]

I've seen this before but unfortunately I think it has to be fixed on the streamline side of things. There is a manual mode where applications directly link against streamline so there's no interface to hook between streamline and the application, so the only way to fix this would be to entirely overhaul RenderDoc's hooking.

Realistically the only reason streamline is doing things this way is specifically to try and defeat function hooks like RenderDoc, as a sort of drm/anti-cheat. If you modify the build to use GetProcAddress like any other normal application things work fine even with the interposer. If your application uses streamline you can make that change locally and it will work normally with RenderDoc.

[thewhitegoatcb]

I doubt that not using GetProcAddress was some sort of anti cheat measure as there's dozens other ways to achieve the same goal and besides Streamline itself being open source and supports plugins.

applications directly link against streamline so there's no interface to hook between streamline and the application

That would be the IAT that you already hooking with other libraries, you can just hook it the same way an treat it as a d3d11/12.dll