NeRF4SeRe: Neural Radiance Fields for Scene Reconstruction
Project of AI3604 Computer Vision, SJTU.
Wei Jiang *
·
Kangrui Mao *
·
Gu Zhang *
·
Haoyu Zhen *
\star = equal contribution
TODO
- installation guideline
- training and testing code for NeRF, INeRF and Instant-NGP
- implementation of NeRFusion
- NICE-SLAM integration
- visualizations
Installation
git clone --recursive https://github.com/NeRF-SeRe/SeRe.git
cd SeRe
-
Environment setup You should install google-sparsehash first following the instructions in torchsparse (e.g.,
sudo apt-get install libsparsehash-dev
on Ubuntu andbrew install google-sparsehash
on Mac OS).Then install the dependencies:
conda create -c conda-forge -n sere python=3.9 pip install -r requirements.txt pip install git+https://github.com/facebookresearch/pytorch3d.git pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0 export PYTHONPATH="/path/to/sere:$PYTHONPATH"
-
Dataset preparation Please refer to https://github.com/cvg/nice-slam to download the Replica Dataset. For SJTUers, you could download it from Replica-JBox.
Then link the dataset
ln -s /path/to/Replica data/replica
Training Pipeline
Adding --device x
to specify the GPU device (default: 0).
-
Vanilla NeRF
python run/main_nerf.py --config configs/nerf/replica/room0.txt
-
Instant-NGP
python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray
-
NeRFusion
python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --fusion_scenes office0 office1 --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 # change fusion_scenes to specify the scenes to be trained
-
Refine camera pose (INeRF)
After you have trained a NeRF model, please put the checkpoints (its name should keep the same as model_name in config) under /ckpts/inerf. Then run:
python run/main_inerf.py --config configs/inerf/replica/room0.txt
Evaluation
If you have already trained a NeRF model, just add --test
to test it. eg.
python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test # write images
python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test --video # generate video
Acknowledgement
For TA
Our latest report and appendices are in docs
.