Nikhil Keetha
·
Jay Karhade
·
Krishna Murthy Jatavallabhula
·
Gengshan Yang
·
Sebastian Scherer
Deva Ramanan
·
Jonathon Luiten
Paper | Video | Project Page
Table of Contents
First, you have to make sure that you have all dependencies in place. The simplest way is to use anaconda.
You can create an anaconda environment called splatam.
conda env create -f environment.yml
conda activate splatamYou can SplaTAM your own environment with an iPhone or LiDAR-equipped Apple device by downloading and using Nerf-capture.
Make sure that your iPhone and PC are connected to the same WiFi network, and then run the following command:
bash bash_scripts/online_demo.bash configs/iphone/online_demo.pyOn the app, keep clicking send for successive frames, and watch SplaTAM's reconstruction visualizations! Here's a cool example result!
You can also run offline optimization for your iPhone captures by first capturing a dataset with the following command:
bash bash_scripts/nerfcapture2dataset.bashThen, perform slam with the following command:
bash bash_scripts/nerfcapture.bash configs/iphone/nerfcapture.pyDownload the data as below, and the data is saved into the ./datasets/Replica folder. Note that the Replica data is generated by the authors of iMAP (but hosted by the authors of NICE-SLAM). Please cite iMAP if you use the data.
bash bash_scripts/download_replica.shbash bash_scripts/download_tum.shDATAROOT is ./data by default. Please change the input_folder path in the scene-specific config files if stored somewhere else on your machine.
Please follow the data downloading procedure on the ScanNet website, and extract color/depth frames from the .sens file using this code.
[Directory structure of ScanNet (click to expand)]
DATAROOT is ./datasets by default. If a sequence (sceneXXXX_XX) is stored in other places, please change the input_folder path in the config file or in the command line.
DATAROOT
└── scannet
└── scene0000_00
└── frames
├── color
│ ├── 0.jpg
│ ├── 1.jpg
│ ├── ...
│ └── ...
├── depth
│ ├── 0.png
│ ├── 1.png
│ ├── ...
│ └── ...
├── intrinsic
└── pose
├── 0.txt
├── 1.txt
├── ...
└── ...
We use the following sequences:
scene0000_00
scene0059_00
scene0106_00
scene0181_00
scene0207_00
Please follow the data downloading and image undistortion procedure on the ScanNet++ website. We use the following sequences:
8b5caf3398
b20a261fdf
For b20a261fdf, we use the first 360 frames.
We use the Replica-V2 to evaluate novel view synthesis, we use the pre-generated replica sequences from vMAP.
For running SplaTAM, we recommend using weights and biases for the logging. This can be turned on by setting the wandb flag to True in the configs file. Also make sure to specify the path wandb_folder. If you don't have a wandb account, first create one. Each scene has a config folder, where the input_folder and output paths need to be specified. Below, we show some example run commands for one scene from each dataset.
To run SplaTAM on the room0 scene, run the following command.
python scripts/slam.py configs/Replica/slam.pyAfter reconstruction, the trajectory error will be evaluated along with the rendering metrics. For other scenes, modify the configs/Replica/slam.py file.
To run SplaTAM on the freiburg1_desk scene, run the following command.
python run.py configs/TUM_RGBD/slam.pyAfter reconstruction, the trajectory error will be evaluated along with the rendering metrics. For other scenes, modify the configs/TUM_RGBD/slam.py file.
To run SplaTAM on the scene0000_00 scene, run the following command.
python run.py configs/ScanNet/slam.pyAfter reconstruction, the trajectory error will be evaluated along with the rendering metrics. For other scenes, modify the configs/ScanNet/slam.py file.
We thank the authors of the following repositories for their open-source code:
- 3D Gaussians
- Dataloaders
- Baselines
If you find our paper and code useful, please cite it as:
@article{keetha2023splatam,
author = {Keetha, Nikhil and Karhade, Jay and Jatavallabhula, Krishna Murthy and Yang, Gengshan and Scherer, Sebastian and Ramanan, Deva and Luiten, Jonathan}
title = {SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM},
journal = {arXiv},
year = {2023},
}