This repository contains the code to generate the object insertion results in paper Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image, CVPR 2020. Please visit our project webpage for more information.
We hope that the code can help reproduce the photorealistic object insertion results in our paper and compare future lighting estimation methods more easily. Please consider citing the paper if you find the code useful. Please contact Zhengqin Li if you have any questions.
The original models were trained by extending the SUNCG dataset with an SVBRDF-mapping. Since SUNCG is not available now due to copyright issues, we are not able to release the original models. Instead, we rebuilt a new high-quality synthetic indoor scene dataset and trained our models on it. We will release the new dataset in the near future. Some object insertion results on real images generated using our network trained on the new dataset are shown below.
To run our code, you will need
- matlab
- python
- Renderer: Please find the renderer from this link, which is an Optix-based renderer implemented for this project.
Please download the data from the link. It includes the inverse rendering predictions of the 20 examples from Garon et al. 2019 dataset (Example 1-20) and 4 examples (demo0-3) downloaded from the Internet.
To reproduce the 20 object insertion results from Garon et al. 2019 dataset.
- Download the inverse rendering predictions from the link mentioned above. Unzip the data in the same directory as the code.
- Open
generateXMLread_ref.m. On line 3 and 4, write down the path to your renderer and your python program. - Run
generateXMLread_ref.m
The program will read the im_*.mat files in each Example* directory to generate the object insertion results.
We also provide the code to generate the object insertion code from scratch. The instructions are as follows:
- Similarly, download the inverse rendering results from this link
- Open
generateXML.m. On line 4 and 5, enter the path to your renderer and your python program. - Several parameters from can be used to adjust the object insertion results
meshNewName: The mesh used for object insertion. We provide two meshes:sphere.objandbunny.obj. You can include your own mesh but you may need to scale and rotate the mesh accordingly so that they will look good.r, g, b: The diffuse color of the inserted object. We set them all to 0.8 in our experiments.roughness: The roughness value of the inserted object. The smaller the roughness value, the more specular the object will be. We set roughness value to 0.2 in all our experiments.scale: The size of the inserted object. The size of the inserted object is decided by the size of the plane you selected from the image (will be explained more detailedly in the following) and the scale parameter.isNewMask: Whether to select new plane or not. Sometimes, after selecting the plane, you may find the rendered object too small or too large. In that case, you can setisNewMaskto befalseand adjustscale, until you get the satisfactory object insertion result.
- Run
generateXML.m:- You will see an image poped out. Choose 4 points on the image as the plane you select. An example is shown below.
- After choosing the 4 point, the image will be shown again with a red square polygon demonstrating the plane you selected. Choose a point inside the polygon as the place where you would like to inser the object.
- Done, the rendered result is shown below. The image will be saved as
im_new.pngin therootdirectory set on line 3.
- You will see an image poped out. Choose 4 points on the image as the plane you select. An example is shown below.
- If you woild like to insert more objects, just change the
originNamefromim.pngtoim_new.png. Repeat the process and you can insert multiple objects at different locations of a single image.
The current code only supports inserting an object on a planar surface. It has not been extended to handle objects in arbitrary 3D geometry yet.