lzhnb / GS-IR

GS-IR: 3D Gaussian Splatting for Inverse Rendering

Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, Kui Jia

Project Page | ArXiv | Paper

Introduction

We present GS-IR that models a scene as a set of 3D Gaussians to achieve physically-based rendering and state-ofthe-art decomposition results for both objects and scenes.

Welcome to our new work Analytic-Splatting. We achieve anti-aliasing and excellent detail fidelity through analytical integral approximation.

Installation

create the basic environment

conda env create --file environment.yml
conda activate gsir

pip install kornia

install some extensions

cd gs-ir && python setup.py develop && cd ..

cd submodules
git clone https://github.com/NVlabs/nvdiffrast
pip install ./nvdiffrast

pip install ./simple-knn
pip install ./diff-gaussian-rasterization # or cd ./diff-gaussian-rasterization && python setup.py develop && cd ../..

Dataset

We evaluate our method on TensoIR-Synthetic and Mip-NeRF 360 datasets. And please visit here for the environment maps. Please refer to the TensoIR for more details about TensoIR-Synthetic and Environment map.

Running

TensoIR-Synthetic

Take the lego case as an example.

Stage1 (Initial Stage)

python train.py \
-m outputs/lego/ \
-s datasets/TensoIR/lego/ \
--iterations 30000 \
--eval

Baking

python baking.py \
-m outputs/lego/ \
--checkpoint outputs/lego/chkpnt30000.pth \
--bound 1.5 \
--occlu_res 128 \
--occlusion 0.25

Stage2 (Decomposition Stage)

python train.py \
-m outputs/lego/ \
-s datasets/TensoIR/lego/ \
--start_checkpoint outputs/lego/chkpnt30000.pth \
--iterations 35000 \
--eval \
--gamma \
--indirect

set --gamma to enable linear_to_sRGB will cause better relighting results but worse novel view synthesis results set --indirect to enable indirect illumination modelling

Evaluation (Novel View Synthesis)

python render.py \
-m outputs/lego \
-s datasets/TensoIR/lego/ \
--checkpoint outputs/lego/chkpnt35000.pth \
--eval \
--skip_train \
--pbr \
--gamma \
--indirect

Evaluation (Normal)

python normal_eval.py \
--gt_dir datasets/TensoIR/lego/ \
--output_dir outputs/lego/test/ours_None

Evaluation (Albedo)

python render.py \
-m outputs/lego \
-s datasets/TensoIR/lego/ \
--checkpoint outputs/lego/chkpnt35000.pth \
--eval \
--skip_train \
--brdf_eval

Relighting

python relight.py \
-m outputs/lego \
-s datasets/TensoIR/lego/ \
--checkpoint outputs/lego/chkpnt35000.pth \
--hdri datasets/TensoIR/Environment_Maps/high_res_envmaps_2k/bridge.hdr \
--eval \
--gamma

set --gamma to enable linear_to_sRGB will cause better relighting results but worse novel view synthesis results

Relighting Evaluation

python relight_eval.py \
--output_dir outputs/lego/test/ours_None/relight/ \
--gt_dir datasets/TensoIR/lego/

Mip-NeRF 360

Take the bicycle case as an example.

Stage1 (Initial Stage)

python train.py \
-m outputs/bicycle/ \
-s datasets/nerf_real_360/bicycle/ \
--iterations 30000 \
-i images_4 \
-r 1 \
--eval

-i images_4 for outdoor scenes and -i images_2 for indoor scenes -r 1 for resolution scaling (not rescale)

Baking

python baking.py \
-m outputs/bicycle/ \
--checkpoint outputs/bicycle/chkpnt30000.pth \
--bound 16.0 \
--occlu_res 256 \
--occlusion 0.4

Stage2 (Decomposition Stage)

python train.py \
-m outputs/bicycle \
-s datasets/nerf_real_360/bicycle/ \
--start_checkpoint outputs/bicycle/chkpnt30000.pth \
--iterations 40000 \
-i images_4 \
-r 1 \
--eval \
--metallic \
--indirect

set --metallic choose to reconstruct metallicness set --gamma to enable linear_to_sRGB will cause better relighting results but worse novel view synthesis results set --indirect to enable indirect illumination modelling

Evaluation

python render.py \
-m outputs/bicycle \
-s datasets/nerf_real_360/bicycle/ \
--checkpoint outputs/bicycle/chkpnt40000.pth \
-i images_4 \
-r 1 \
--eval \
--skip_train \
--pbr \
--metallic \
--indirect

set --gamma to enable linear_to_sRGB will cause better relighting results but worse novel view synthesis results

Relighting

python relight.py \
-m outputs/bicycle \
-s datasets/nerf_real_360/bicycle/ \
--checkpoint outputs/bicycle/chkpnt40000.pth \
--hdri datasets/TensoIR/Environment_Maps/high_res_envmaps_2k/bridge.hdr \
--eval \
--gamma

set --gamma to enable linear_to_sRGB will cause better relighting results but worse novel view synthesis results

Acknowledge

• gaussian-splatting
• nvdiffrast
• nvdiffrec

Citation

If you find this work useful in your research, please cite:

@article{liang2023gs,
  title={Gs-ir: 3d gaussian splatting for inverse rendering},
  author={Liang, Zhihao and Zhang, Qi and Feng, Ying and Shan, Ying and Jia, Kui},
  journal={arXiv preprint arXiv:2311.16473},
  year={2023}
}