Animatable Neural Radiance Fields from Monocular RGB Videos
Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, Huchuan Lu
More demos please see Demos.
- pyrender
- Trimesh
- PyMCubes
Run the following code to install all pip packages:
pip install -r requirements.txt
To install KNN_CUDA, we provide two ways:
- from source
git clone https://github.com/unlimblue/KNN_CUDA.git cd KNN_CUDA make && make install
- from wheel
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
To download the SMPL model go to this (male, female and neutral models).
Place them as following:
smplx
└── models
└── smpl
├── SMPL_FEMALE.pkl
├── SMPL_MALE.pkl
└── SMPL_NEUTRAL.pkl
- prepare images and smpls
python -m tools.people_snapshot --data_root ${path_to_people_snapshot_datasets} --people_ID male-3-casual --gender male --output_dir data/
- prepare template
python -m tools.prepare_template --data_root data/ people_ID male-3-casual --model_type smpl --gender male --model_path ${path_to_smpl_models}
- Training on the training frames
python train.py --cfg_file configs/people_snapshot/male-3-casual.yml
- Finetuning the smpl params on the testing frames
python train.py --cfg_file configs/people_snapshot/male-3-casual_refine.yml train.ckpt_path checkpoints/male-3-casual/last.ckpt
We provide the preprocessed data and pretrained models at Here
python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt
python extract_mesh.py --ckpt_path checkpoints/male-3-casual/last.ckpt
python novel_pose.py --ckpt_path checkpoints/male-3-casual/last.ckpt
python test.py --ckpt_path checkpoints/male-3-casual_refine/last.ckpt --vis
If you find the code useful, please cite:
@misc{chen2021animatable,
title={Animatable Neural Radiance Fields from Monocular RGB Videos},
author={Jianchuan Chen and Ying Zhang and Di Kang and Xuefei Zhe and Linchao Bao and Xu Jia and Huchuan Lu},
year={2021},
eprint={2106.13629},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Parts of the code were based on from kwea123's NeRF implementation: https://github.com/kwea123/nerf_pl. Some functions are borrowed from PixelNeRF https://github.com/sxyu/pixel-nerf