Strivec: Sparse Tri-Vector Radiance Fields

Overal Instruction

We build the initial geometry with the 1st stage of DVGO in our implementation by default, which is use_geo = -1 in config files.
The geometry can be either initialized online (by default) or from other sources in .txt form, which can be enabled with use_geo = 1 and pointfile = /your/file.txt in config files.
You may ignore preprosessing folder, which is initially for our ealy trying and not used here.
You may refer to the comments in ./configs/synthetic-nerf/default/chair.txt for the usage of hyperparameters.

For Synthetic-NeRF dataset, we provide the initial geometry from DVGO (which is the default one in our implementation) and from MVS. Feel free to try both (e.g., use_geo = 1 and pointfile = /your/mvs_file.txt) to see the comparison.

For Scannet dataset, we use the initial geometry provided by the dataset itself. We convert the original .ply file into .txt and you may download from here.

Installation

Requirements

All the codes are tested in the following environment:

Linux 18.04+
Python 3.6+
PyTorch 1.10+
CUDA 10.2+

Data Preparation

And the layout should look like this:

Strivec
├── data
│   ├── nerf_synthetic
    │   │   |──default
    │   │   │   |──chair
    │   │   │   │──drums
    │   │   │   |──...
    │   │   |──local_vm
    │   │   │   |──chair
    │   │   │   │──drums
    │   │   │   |──...
    ├── scene0101_04 (scannet)
    │   │   │──exported
    │   │   │──scene0101_04_2d-instance-filt.zip
    │   │   │──...
    ├── scene0241_01 (scannet)
    │   │   │──exported
    │   │   │──scene0241_01_2d-instance-filt.zip
    │   │   │──...
    ├── TanksAndTemple
    │   │   │──Barn
    │   │   │──Caterpillar
    │   │   │──...
    ├── 360 (Mip-NeRF360)
    │   │   │──garden
    │   │   │──room
    │   │   │──...

Training & Evaluation

We not only provide the training and evaluation code to reproduce the results in the paper, but also the code of ablation that uses local VM tensors instead of local CP tensors (results are here).

# hierachical Strivec, without rotation (grid aligned)
python train_hier.py --config ./configs/synthetic-nerf/default/chair.txt

# local VM tensors instead of local CP tensors
train_dbasis.py --config ./configs/synthetic-nerf/local_vm/chair.txt

Visualization

We visualize the local tensors of different scales into ./log/your_scene/rot_tensoRF/0_lvl_k.ply, where k is the kth scale.

Citation

If you find our code or paper helps, please consider citing:

@INPROCEEDINGS{gao2023iCCV,
  author = {Quankai Gao and Qiangeng Xu and Hao su and Ulrich Neumann and Zexiang Xu},
  title = {Strivec: Sparse Tri-Vector Radiance Fields},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year = {2023}
}

Xharlie / Strivec