SJoJoK / VGOS

[IJCAI 2023] Official code for the paper "VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs".

VGOS

Paper | Supplementary Material

Official code for the IJCAI 2023 paper "VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs".

Installation

git clone [e-mail]:SJoJoK/VGOS.git
cd VGOS
pip install -r requirements.txt

Pytorch and torch_scatter installation is machine dependent, please install the correct version for your machine.

We conduct our experiments with torch 1.13.0+cu117 and torch-scatter 2.1.0+pt113cu117 on a single NVIDIA GeForce RTX 3090 GPU (24 GB).

Directory structure for the datasets

(click to expand)

data
├── nerf_synthetic     # Link: https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1
│   └── [chair|drums|ficus|hotdog|lego|materials|mic|ship]
│       ├── [train|val|test]
│       │   └── r_*.png
│       └── transforms_[train|val|test].json
│
└── nerf_llff_data     # Link: https://drive.google.com/drive/folders/14boI-o5hGO9srnWaaogTU5_ji7wkX2S7
    └── [fern|flower|fortress|horns|leaves|orchids|room|trex]
        ├── poses_bounds.npy
        └── [images_2|images_4]

GO

• Training

  The PSNR, SSIM and LPIPS of the testset will be evaluated after training.

  $ python run.py --config configs/llff_3v/room.py --render_test_get_metric

• Render w/o re-training

  $ python run.py --config configs/llff_3v/room.py --render_only --render_video

• Re-conduct the experiments in the paper

  $ zsh train_script/llff/llff_3v.sh
  $ zsh train_script/blender/blender_4v.sh

  Feel free to modify the train scripts to suit your own needs, such as adding experiments under different training views or tuning parameters (please refer to the comments in configs/default.py for more information about the parameters).

  Please note that the random selection of training views may result in fluctuations in performance. Therefore, the metrics presented for comparisons in our paper are the average scores of five experiments with different viewpoint samples. It is common to observe results that are slightly different, either higher or lower, than those presented in the paper.

Acknowledgement

The code is heavily based on DVGOv2 implementation, and the computation of depth smoothness loss is modified from RegNeRF. Thanks for sharing!

Citation

If you find this repo is helpful, please cite:

@inproceedings{ijcai2023p157,
  title     = {VGOS: Voxel Grid Optimization for View Synthesis from Sparse Inputs},
  author    = {Sun, Jiakai and Zhang, Zhanjie and Chen, Jiafu and Li, Guangyuan and Ji, Boyan and Zhao, Lei and Xing, Wei},
  booktitle = {Proceedings of the Thirty-Second International Joint Conference on
               Artificial Intelligence, {IJCAI-23}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Edith Elkind},
  pages     = {1414--1422},
  year      = {2023},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2023/157},
  url       = {https://doi.org/10.24963/ijcai.2023/157},
}