Official implementation of Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion
Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, Yueqi Duan
Paper | Project page | Data
Physics3D is a unified simulation-rendering pipeline based on 3D Gaussians, which learn physics dynamics from video diffusion model.
The repo is still being under construction, thanks for your patience.
- Training code release.
- Synthetic data release.
- Detailed tutorial.
- Detailed local demo.
conda create -n Physics3D python=3.9
conda activate Physics3D
pip install -r requirements.txt
git clone https://github.com/graphdeco-inria/gaussian-splatting
pip install -e gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install -e gaussian-splatting/submodules/simple-knn/
-
Download the Gaussian models from OneDrive. You can also load your own 3D Gaussian pre-trained models to this pipeline following gaussian-splatting. For the setting details of physical configs, you can refer to PhysGaussian.
Physics3D ├──model ├── ball/ ├──config ├── ball_config.json -
We support using text-to-video (ModelScope) diffusion models to guide the optimization of physical parameters. You can use the following command:
python simulation.py --model_path ./model/ball/ --prompt "a basketball falling down" --output_path ./output --physics_config ./config/ball_config.json
-
Parameter initialization that aligns with physical facts can significantly accelerate the convergence of Physics3D and improve training effectiveness.
-
For some high-frequency elastic objects, simulation effectiveness can be enhanced by increasing particle density.
We have intensively borrowed code from the following repositories. Many thanks to the authors for sharing their code.
- DreamPhysics
- threestudio and its extension for Animate124.
- warp-mpm
- PhysGaussian
We have also used open-source datasets from the following repositories.
- PhysDreamer
- BlenderKit for free models and BlenderNeRF for synthetic NeRF datasets within Blender
If you found Physics3D helpful, please cite our report:
@article{liu2024physics3d,
title={Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion},
author={Liu, Fangfu and Wang, Hanyang and Yao, Shunyu and Zhang, Shengjun and Zhou, Jie and Duan, Yueqi},
journal={arXiv preprint arXiv:2406.04338},
year={2024}
}If you have any question about this project, please feel free to contact liuff23@mails.tsinghua.edu.cn or hanyang-21@mails.tsinghua.edu.cn.