Prerequisites: run on a Linux system with a recent NVidia RTX GPU with at least 8 GB of VRAM. Git must be installed.
- Activate a Conda environment with PyTorch that supports Nerfstudio
- Possibly required, depending on your environment:
conda install -c conda-forge gcc=12.1.0
- Run
./scripts/install.sh
(see steps within if something goes wrong)
The inputs directly trainable with our fork of Nerfstudio are stored in data/inputs-processed
folder.
Its subfolders are called "datasets" in these scripts.
The data can be automatically downloaded by first installing: pip install unzip
and then running
python download_data.py --dataset synthetic
# or 'sai' for processed real world smartphone data
The data folder structure is as follows:
<3dgs-deblur>
|---data
|---inputs-processed
|---colmap-sai-cli-vels-blur-scored/
|---iphone-lego1
|---images
|---image 0
|---image 1
|---...
|---sparse_pc.ply
|---transforms.json
|---...
|---synthetic-mb
|---cozyroom
|---images
|---image 0
|---image 1
|---...
|---sparse_pc.ply
|---transforms.json
|---...
|---...
|---...
Example: List trainable variants for the synthetic-mb
dataset:
python train.py --dataset=synthetic-mb
Train a single variant
python train.py --dataset=synthetic-mb --case=2
Common useful options:
--dry_run
--preview
(show Viser during training)
Second pass for pose_opt
variants: After training all the relevant pose_opt
variants with the --train_all
flag, e.g.,
python train.py --dataset=synthetic-posenoise --train_all --case=2
run, e.g.,
python combine.py \
--dataset=synthetic-posenoise \
--pose_opt_pass_dir data/outputs/synthetic-posenoise/pose_opt-train_all \
--case=1
This creates new datasets in input-processed/
with 2nd-pass
in their name. These can be trained again with train.py
:
python train.py --dataset=synthetic-posenoise-2nd-pass --case=1
If all relevant first pass runs have been completed, you can also run ./scripts/process_2nd_pass.sh
,
which creates all the corresponding second pass datasets.
The method can be also be used with custom data recorded using the Spectacular Rec app (v1.0.0+ for Android, v1.2.0+ for iOS).
First, download and extract a recording created using the app, e.g., /PATH/TO/spectacular-rec-MY_RECORDING
.
iOS cases (short rolling shutter read-out): Process as
./scrpts/process_and_train_sai_custom_mb.sh /PATH/TO/spectacular-rec-MY_RECORDING
Android (long rolling-shutter readout). The recommended mode is:
./scripts/process_and_train_sai_custom_mbrs_pose_opt.sh
See the contents of the script for more details.
Additionally, any folder of the form data/inputs-processed/CASE
can be trained directly with Nerfstudio
using the ns-train splatfacto --data data/inputs-processed/CASE ...
. Use --help
and see train.py
for
the recommended parameters.
Comparison video To train a custom recording with and without motion blur compensation and render a video comparing the two, use this script:
./scripts/render_and_train_comparison_sai_custom_mb.sh /PATH/TO/spectacular-rec-MY_RECORDING
Results are written to data/outputs/
by dataset. You can also run these on another machine
and download these results on your machine. All of the below commands should then work for
locally examining the results.
List all numeric results
python parse_outputs.py
... or export to CSV
python parse_outputs.py -f csv > data/results.csv
Off-the-shelf:
- Viser:
ns-viewer --load-config outputs/DATASET/VARIANT/splatfacto/TIMESTAMP/config.yml
(show actual results) - Tensorboard:
tensorboard --logdir outputs/DATASET/VARIANT/splatfacto/TIMESTAMP
(prerequisitepip install tensorboard
)
Custom:
- Created by
train.py --render_images ...
: Renders of evaluation images and predictions are available inoutputs/DATASET/VARIANT/splatfacto/TIMESTAMP
(/renders
, or/demo_video*.mp4
ifrender_video.py
has been run, see below) - Demo videos: see
render_video.py
andscripts/render_and_combine_comparison_video.sh
This method also creates the extra variants discussed in the appendix/supplementary material of the paper.
Synthetic data: Download and process as
python download_data.py --dataset synthetic-raw
python process_synthetic_inputs.py
Smartphone data: Download as:
python download_data.py --dataset sai-raw
and then process with the following steps
-
Process and convert using the Spectacular AI SDK to get VIO velocity and pose estimates (
--preview
is optional). Creates a dataset calledsai-cli
:python process_sai_inputs.py --preview
-
Run COLMAP. NOTE: this has some level of randomness. COLMAP may or may not fail for some of the sequences, but should eventyally succeed for the included sequences after a few retries. Run until ALL sequences succeed. See also the
python run_colmap.py
to list cases andpython run_colmap.py --case=N
to run a specific case. Creates a dataset calledcolmap-sai-cli-imgs
.python run_colmap.py all
-
Create the other variants by combining and augmenting the above two by running:
./scripts/create_smartphone_variants.sh # or # EXTRA_VARIANTS=ON ./scripts/create_smartphone_variants.sh
Note: all the components in this pipeline are not guaranteed to be deterministic, especially when executed on different machines.
The code in this repository (except the gh-pages
website branch) is licensed under Apache 2.0.
See LICENSE
and NOTICE
files for more information.
For the source code of the website and its license, see the gh-pages
branch.
The licenses of the datasets (CC BY-SA 4.0 & CC BY 4.0) are detailed on the Zenodo pages.