A collaboration friendly studio for NeRFs
About
Nerfstudio provides a simple API that allows for a simplified end-to-end process of creating, training, and testing NeRFs. The library supports a more interpretable implementation of NeRFs by modularizing each component. With more modular NeRFs, we hope to create a more user-friendly experience in exploring the technology. Nerfstudio is a contributor-friendly repo with the goal of building a community where users can more easily build upon each other's contributions.
It’s as simple as plug and play with nerfstudio!
We are committed to providing learning resources to help you understand the basics of (if you're just getting started), and keep up-to-date with (if you're a seasoned veteran) all things NeRF. As researchers, we know just how hard it is to get onboarded with this next-gen technology. So we're here to help with tutorials, documentation, and more!
Have feature requests? Want to add your brand-spankin'-new NeRF model? Have a new dataset? We welcome any and all contributions! Please do not hesitate to reach out to the nerfstudio team with any questions via Discord.
We hope nerfstudio enables you to build faster 🔨 learn together 📚 and contribute to our NeRF community 💖.
Quickstart
The quickstart will help you get started with the default vanilla NeRF trained on the classic Blender Lego scene. For more complex changes (e.g., running with your own data/setting up a new NeRF graph, please refer to our references.
1. Installation: Setup the environment
Prerequisites
CUDA must be installed on the system. This library has been tested with version 11.3. You can find more information about installing CUDA here
Create environment
Nerfstudio requires python >= 3.7. We recommend using conda to manage dependencies. Make sure to install Conda before proceeding.
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pipDependencies
Install pytorch with CUDA (this repo has been tested with CUDA 11.3) and tiny-cuda-nn
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torchInstalling nerfstudio
Easy option:
pip install nerfstudioIf you would want the latest and greatest:
git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
pip install --upgrade pip setuptools
pip install -e .2. Setting up the data
Download the original NeRF Blender dataset. We support the major datasets and allow users to create their own dataset, described in detail here.
ns-download-data --dataset=blender
ns-download-data --dataset=nerfstudio --capture=poster2.x Using custom data
If you have custom data in the form of a video or folder of images, we've provided some COLMAP and FFmpeg scripts to help you process your data so it is compatible with nerfstudio.
After installing both software, you can process your data via:
ns-process-data {video,images,insta360} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR}
# Or if you're on a system without an attached display (i.e. colab):
ns-process-data {video,images,insta360} --data {DATA_PATH} --output-dir {PROCESSED_DATA_DIR} --no-gpu3. Training a model
To run with all the defaults, e.g., vanilla NeRF method with the Blender Lego image
# To see what models are available.
ns-train --help
# To see what model-specific cli arguments are available.
ns-train nerfacto --help
# Run with nerfacto model.
ns-train nerfacto
# We provide support for other models. E.g., to run instant-ngp.
ns-train instant-ngp
# To train on your custom data.
ns-train nerfacto --data {PROCESSED_DATA_DIR}3.x Training a model with the viewer
You can visualize training in real-time using our web-based viewer.
Make sure to forward a port for the websocket to localhost. The default port is 7007, which you should expose to localhost:7007.
# with the default port
ns-train nerfacto --vis viewer
# with a specified websocket port
ns-train nerfacto --vis viewer --viewer.websocket-port=7008
# port forward if running on remote
ssh -L localhost:7008:localhost:7008 {REMOTE HOST}For more details on how to interact with the visualizer, please visit our viewer walk-through.
4. Rendering a trajectory during inference
After your model has trained, you can headlessly render out a video of the scene with a pre-defined trajectory.
# assuming previously ran `ns-train nerfacto`
ns-render --load-config=outputs/data-nerfstudio-poster/nerfacto/{TIMESTAMP}/config.yml --traj=spiral --output-path=renders/output.mp4Learn More
And that's it for getting started with the basics of nerfstudio.
If you're interested in learning more on how to create your own pipelines, develop with the viewer, run benchmarks, and more, please check out some of the quicklinks below or visit our documentation directly.
| Section | Description |
|---|---|
| Documentation | Full API documentation and tutorials |
| Viewer | Home page for our web viewer |
| 🎒 Educational | |
| Model Descriptions | Description of all the models supported by nerfstudio and explanations of component parts. |
| Component Descriptions | Interactive notebooks that explain notable/commonly used modules in various models. |
| 🏃 Tutorials | |
| Getting Started | A more in-depth guide on how to get started with nerfstudio from installation to contributing. |
| Using the Viewer | A quick demo video on how to navigate the viewer. |
| Using Record3D | Demo video on how to run nerfstudio without using COLMAP. |
| 💻 For Developers | |
| Creating pipelines | Learn how to easily build new neural rendering pipelines by using and/or implementing new modules. |
| Creating datasets | Have a new dataset? Learn how to run it with nerfstudio. |
| Contributing | Walk-through for how you can start contributing now. |
| 💖 Community | |
| Discord | Join our community to discuss more. We would love to hear from you! |
| Follow us on Twitter @nerfstudioteam to see cool updates and announcements | |
| TikTok | Coming soon! Follow us on TikTok to see some of our fan favorite results |
Supported Features
We provide the following support structures to make life easier for getting started with NeRFs. For a full description, please refer to our features page.
If you are looking for a feature that is not currently supported, please do not hesitate to contact the Nerfstudio Team on Discord!
- 🔎 Web-based visualizer that allows you to:
- Visualize training in real-time + interact with the scene
- Create and render out scenes with custom camera trajectories
- View different output types
- And more!
- ✏️ Support for multiple logging interfaces (Tensorboard, Wandb), code profiling, and other built-in debugging tools
- 📈 Easy-to-use benchmarking scripts on the Blender dataset
- 📱 Full pipeline support (w/ Colmap or Record3D) for going from a video on your phone to a full 3D render.
Built On
- Easy-to-use config system
- Developed by Brent Yi
- Library for accelerating NeRF renders
- Developed by Ruilong Li
Citation
If you use this library or find the documentation useful for your research, please consider citing:
@misc{nerfstudio,
title={Nerfstudio: A Framework for Neural Radiance Field Development},
author={Matthew Tancik*, Ethan Weber*, Evonne Ng*, Ruilong Li, Brent Yi,
Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi,
Abhik Ahuja, David McAllister, Angjoo Kanazawa},
year={2022},
url={https://github.com/nerfstudio-project/nerfstudio},
}