c7w / renderdoc_for_game_data

Access data with annotations from GTA5 using renderdoc

RenderDoc for Game data

This repo aims to acquire data from commerical games for various vision tasks whose data is difficulty to colloect. Before the start, please get familiar with the great game debugging tool RenderDoc.

What can you get

With this codebase, you can get various intermediate outputs of the rendering pipeline of a game, including depth, meshes and textures. Below are some examples from the commercial game GTAV (Grand Theft Auto V). Note that Rockstar Games, the publisher of GTAV, allows non-commerical use of footage from the game as long as certain conditions are met. See the policy.

Various info in G-Buffer

You may get the diffuse map, normal map, specular map, irradiance map and depth map as follows.

Final Output	Depth Map	Diffuse Map
Normal Map	Irradiance Map	Specular Map

HDR and LDR image pairs

For someone who are interested in data-driven tone mapping and reverse tone mapping, it is a good choice to get HDR and LDR image pairs from the game.

LDR Output	HDR Output

Semantic label

With mesh data and their various coordinates, you can even get the masks for amodal isntance semantic segementation where ostackled parts of the object should be labeled in the mask.

Original Image	Amodel Mask

Foggy image simulation

With images and corresponding depth, you can get clear and foggy image pairs to faciliate your defogging researches.

Original Image	Simulated Foggy Image
Original Image	Simulated Dust Image

Customization on your own

To begain with, you should get to know the rendering pipeline of your target game. For GTAV, you may find useful information in this Link. Then, compile the RenderDoc. Since I have just modified functions of capture and saving to save the depth maps, if you don't need those you may use the original RenderDoc. After that, capture frames you need. Finally, run the processing code in scripts/python in cmd (not the shell of RenderDoc).

Run extract_data_for_GTA5.py to get LDR, HDR, and depth maps. You may modify the variables log_file_root and save_roots on your own settings. If something's wrong with capAPIForGTAV.py, there may be a update in the rendering pipeline. In this case, you should locate the required data in the pipeline using the GUI of RenderDoc and modify the code accordingly.

Citation

If you find this project useful for your research, please consider citting the following BibTeX entry.

@article{zhang2021simulation,
  title={Simulation of Atmospheric Visibility Impairment},
  author={Zhang, Lin and Zhu, Anqi and Zhao, Shiyu and Zhou, Yicong},
  journal={IEEE Transactions on Image Processing},
  volume={30},
  pages={8713--8726},
  year={2021},
  publisher={IEEE}
}