InNeRF360

'Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields' (CVPR 2024)

Project | arXiv

Dongqing Wang, Tong Zhang, Alaa Abboud, Sabine Süsstrunk

Abstract: We propose InNeRF360, an automatic system that accurately removes text-specified objects from 360-degree Neural Radiance Fields (NeRF). The challenge is to effectively remove objects while inpainting perceptually consistent content for the missing regions, which is particularly demanding for existing NeRF models due to their implicit volumetric representation. Moreover, unbounded scenes are more prone to floater artifacts in the inpainted region than frontal-facing scenes, as the change of object appearance and background across views is more sensitive to inaccurate segmentations and inconsistent inpainting. With a trained NeRF and a text description, our method efficiently removes specified objects and inpaints visually consistent content without artifacts. We apply depth-space warping to enforce consistency across multiview text-encoded segmentations, and then refine the inpainted NeRF model using perceptual priors and 3D diffusion-based geometric priors to ensure visual plausibility. Through extensive experiments in segmentation and inpainting on 360-degree and frontal-facing NeRFs, we show that our approach is effective and enhances NeRF's editability.

If you find this project useful for your research, please cite:

@InProceedings{wang2024innerf360,
    author={Wang, Dongqing and Zhang, Tong and Abboud, Alaa and S{\"u}sstrunk, Sabine},
    title     = {{InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fields}},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024}
}

Installation

Setup conda

conda create --name innerf360 -y python=3.9
conda activate innerf360
python -m pip install --upgrade pip

Install Nerfstudio and dependencies.

cd nerfstudio
pip install -e .
pip install torch==1.13.1 torchvision functorch --extra-index-url https://download.pytorch.org/whl/cu117
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Install binvox

mkdir bins
cd bins
wget -O binvox https://www.patrickmin.com/binvox/linux64/binvox?rnd=16811490753710
cd ../
chmod +x bins/binvox

Install InNeRF360

cd ../
git clone https://github.com/IVRL/InNeRF360.git
cd InNeRF360

Project Organization

root
├── nerfstudio               # installation of NeRFStudio
└── InNeRF360                # github repo fils. Model details to be released!
    ├── data
        └── ShapeNetCore.v2  # ShapeNet data
    ├── data_modules
    ...
    └── bins
        └── binvox           # binvox to voxelize cubes
└── docs                     # website

3D diffusion model

Our geometric prior is trained on sampling cubes from ShapeNet meshes. To download the ShapeNet dataset, login or create an account at https://shapenet.org and then download the ShapeNetCore.v2 dataset.

We will release training details soon, in the meantime, we provide a trained pretrained checkpoint for our model.

Dataset Preparation

We perform our experiments on NeRFStudio datasets, which can be downloaded from the website as follows:

ns-download-data nerfstudio --capture-name nerfstudio-dataset

Inference

Our code is build upon NeRFbusters from a specific branch of the original NeRFStudio model. It is thus not feasible to use the default ns-viewer to inspect the trained results.

For inspection and comparison purposes, we retrain our checkpoints to be compatible with default viewer, which can be found here.

(TODO) Training

Acknowledgement

This project is based on Nerfstudio and Nerfbusters. We thank the incredible codebases of our brilliant prior works.

IVRL / InNeRF360