We present MovingParts, a NeRF-based method for dynamic scene reconstruction and part discovery. We consider motion as an important cue for identifying parts, that all particles on the same part share the common motion pattern. From the perspective of fluid simulation, existing deformation-based methods for dynamic NeRF can be seen as parameterizing the scene motion under the Eulerian view, i.e., focusing on specific locations in space through which the fluid flows as time passes. However, it is intractable to extract the motion of constituting objects or parts using the Eulerian view representation. In this work, we introduce the dual Lagrangian view and enforce representations under the Eulerian/Lagrangian views to be cycle-consistent. Under the Lagrangian view, we parameterize the scene motion by tracking the trajectory of particles on objects. The Lagrangian view makes it convenient to discover parts by factorizing the scene motion as a composition of part-level rigid motions. Experimentally, our method can achieve fast and high-quality dynamic scene reconstruction from even a single moving camera, and the induced part-based representation allows direct applications of part tracking, animation, 3D scene editing, etc.
The code is tested under Python 3.8.13, CUDA 10.2, and PyTorch 1.11.0.
git clone https://github.com/SilenKZYoung/MovingParts.git
cd MovingParts
conda env create -f environment.yml
Please download the D-NeRF 360 synthetic dataset from dropbox and put it under the ./data. See the following directory structure for an example:
├── data
│ ├── mutant
│ ├── standup
│ ├── ...
# Train
python train.py --logdir ./log --data_dir ./data/XXX/
# Test
python train.py --ckpt ./log/XXX/XXX.th --data_dir ./data/XXX/
MIT License