apollohuang1/pointrix

Pointrix

A differentiable point-based rendering library.
Document🏠 | Paper(Comming soon)📄 | DPTR Backend🌐

Pointrix is a differentiable point-based rendering library which has following properties:

Highly Extensible:
- Python API
- Modular design for both researchers and beginners
- Implementing your own method without touching CUDA
Powerful Backend:
- CUDA Backend
- Forward Anything: rendering image, depth, normal, optical flow, etc.
- Backward Anything: optimizing even intrinsics and extrinsics.
Rich Features:
- 3D Reconstruction: Vanilla 3DGS (Siggraph 2023), Relightable 3DGS(WIP)
- 4D Reconstruction: Deformable 3DGS (CVPR 2024), Gaussian-Flow (CVPR 2024, WIP)
- 3D Generation: GSGen (CVPR 2024, WIP), DreamGaussian (ICLR 2024, TBD), LGM (arXiv 2024, TBD)
- 4D Generation: STAG4D (arXiv 2024, WIP), DreamGaussian4D (arXiv 2023, TBD)
- ...

Comparation with original 3D gaussian code

nerf_synthetic dataset (PSNR)

Method	lego	chair	ficus	drums	hotdog	ship	materials	mic	average
Pointrix	35.84	36.12	35.02	26.18	37.81	30.98	29.95	35.34	33.40
original	35.88	35.92	35.00	26.21	37.81	30.95	30.02	35.35	33.39

we obtain the result of 3D gaussian code by running following command in their repository.

 python train.py -s nerf_synthetic_root --eval -w

Quickstart

Installation

Clone pointrix:

git clone https://github.com/pointrix-project/pointrix.git  --recursive

Create a new conda environment with pytorch:

conda create -n pointrix python=3.9
conda activate pointrix
conda install pytorch==2.1.1 torchvision==0.16.1 pytorch-cuda=12.1 -c pytorch -c nvidia

Install dependencies:

# Install simple knn
cd pointrix
cd submodules/simple-knn
pip install -e .

Install Pointrix and DPTR:

cd ../..
cd dptr
pip install .

cd ..
pip install -r requirements.txt
pip install -e .

Train Your First 3D Gaussian

NeRF-Lego (NeRF-Synthetic format dataset)

Download the lego data:

wget http://cseweb.ucsd.edu/~viscomp/projects/LF/papers/ECCV20/nerf/nerf_example_data.zip

Run the following (with adjusted data path):

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/nerf_dptr.yaml trainer.dataset.data_path=your_data_path

# you can also run the following for the original 3DGS kernel
# python launch.py --config ./configs/nerf.yaml trainer.dataset.data_path=your_data_path

if you want to test the model:

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/nerf_dptr.yaml trainer.training=False trainer.dataset.data_path=your_data_path trainer.test_model_path=your_model_path

Mip-NeRF 360 (Colmap format dataset)

Download the data and run:

cd pointrix/projects/gaussian_splatting
# For Mip-NeRF 360 data which have high-res images and need to downsample.
python launch.py --config ./configs/colmap_dptr.yaml trainer.dataset.data_path=your_data_path  trainer.dataset.scale=0.25

# you can also run the following for the original 3DGS kernel
# python launch.py --config ./configs/colmap.yaml trainer.dataset.data_path=your_data_path

For other colmap dataset which do not need to downsample:

python launch.py --config ./configs/colmap_dptr.yaml trainer.dataset.data_path=your_data_path  trainer.dataset.scale=1.0

Advanced Approaches

Deformable 3D Gaussian

Download the iphone dataset and put it in your folder: https://drive.google.com/drive/folders/1cBw3CUKu2sWQfc_1LbFZGbpdQyTFzDEX
Run the following command to train the model (...data path in the config file...):

cd pointrix/projects/deformable_gaussian
python launch.py --config deform.yaml

Gaussian-Flow (WIP)

GSGen (WIP)

STAG4D (WIP)

Relightable 3DGS (WIP)

Release Plans

GUI for visualization (this week).
Implementataion of Gaussian-Flow (CVPR 2024) (this week).
Implementataion of GSGen with MVDream SDS (this week).
Camera optimization (this week).
Introduction video
Implementataion of Relightable Gaussian (arXiv 2023).

Welcome to discuss with us and submit PR on new ideas and methods.

Contributors

Made with contrib.rocks.

apollohuang1 / pointrix