GaussianCube: Structuring Gaussian Splatting for 3D Generative Modeling using Optimal Transport

By Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, Baining Guo.

Abstract

In this work, we introduce GaussianCube, a novel representation crafted for 3D generative modeling. Recent advances in Gaussian Splatting have demonstrated considerable improvements in expressiveness and efficiency. However, its unstructured nature poses a significant challenge to its deployment in generative applications. To fully unleash the potential of Gaussian Splatting in generative modeling, we present GaussianCube, a powerful and efficient spatial-structured representation for 3D generation. We propose to perform densification-constrained fitting to obtain fix-length Gaussians for each 3D asset, and then systematically arrange the Gaussians into the structured voxel grid via Optimal Transport. Furthermore, we perform generative modeling on the proposed GaussianCube using 3D diffusion models. The spatial coherence of GaussianCube allows us to utilize a standard 3D U-Net as our diffusion network without elaborate design. Extensive experiments conducted on ShapeNet and challenging large-vocabulary OmniObject3D show the superiority to prior works, underscoring its potential as a versatile 3D representation.

Environment Setup

We recommend Linux for performance and compatibility reasons. We use conda to manage the environment. Please install conda from here if you haven't done so.

git clone https://github.com/GaussianCube
cd GaussianCube
conda env create -f environment.yml
conda activate gaussiancube

Model Download

Please download model checkpoints and dataset statistics (pre-computed mean and sta files) from the following links:

Huggingface

Model	Task	Download
OmniObject3D	Class-conditioned Generation	🤗 Hugging Face
ShapeNet Car	Unconditional Generation	🤗 Hugging Face
ShapeNet Chair	Unconditional Generation	🤗 Hugging Face

Inference

Class-conditioned Generation on OmniObject3D

To inference pretrained model of OmniObject3D, save the downloaded model checkpoint and dataset statistics to ./OmniObject3D/, then run:

python inference.py --exp_name /tmp/OmniObject3D_test --config configs/omni_class_cond.yml  --rescale_timesteps 300 --ckpt ./OmniObject3D/OmniObject3D_ckpt.pt  --mean_file ./OmniObject3D/mean.pt --std_file ./OmniObject3D/std.pt  --bound 1.0 --num_samples 10 --render_video --class_cond

Unconditional Generation on ShapeNet

To inference pretrained model of ShapeNet Car, save the downloaded model checkpoint and dataset statistics to ./shapenet_car/, then run:

python render_inference.py --exp_name /tmp/shapenet_car_test --config configs/shapenet_uncond.yml  --rescale_timesteps 300 --ckpt ./shapenet_car/shapenet_car_ckpt.pt  --mean_file ./shapenet_car/mean.pt  --std_file ./shapenet_car/std.pt  --bound 0.45 --num_samples 10 --render_video

To inference pretrained model of ShapeNet Chair, save the downloaded model checkpoint and dataset statistics to ./shapenet_chair/, then run:

python inference.py --exp_name /tmp/shapenet_chair_test --config configs/shapenet_uncond.yml  --rescale_timesteps 300 --ckpt ./shapenet_chair/shapenet_chair_ckpt.pt  --mean_file ./shapenet_chair/mean.pt  --std_file ./shapenet_chair/std.pt  --bound 0.35 --num_samples 10 --render_video

Acknowledgement

This codebase is built upon the improved-diffusion, thanks to the authors for their great work.

Citation

If you find this work useful, please consider citing:

@article{zhang2024gaussiancube,
  title={GaussianCube: Structuring Gaussian Splatting using Optimal Transport for 3D Generative Modeling},
  author={Zhang, Bowen and Cheng, Yiji and Yang, Jiaolong and Wang, Chunyu and Zhao, Feng and Tang, Yansong and Chen, Dong and Guo, Baining},
  journal={arXiv preprint arXiv:2403.19655},
  year={2024}
}

Todo

Release the inference code.
Release all pretrained models.
Release the data fitting code.
Release the diffusion training code.

zoonono / GaussianCube