Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis (CVPR 2022)
Project page | Video | Paper
Installation and Data Preparation
Please see INSTALL.md for installation and data preparation.
Run the code on ZJU-MoCap
Take the subject "313" as an example.
Download the pre-trained model
For a quick start, you can download the pretrained model from Google Drive (to be updated), and put it to $ROOT/data/trained_model/sa-nerf/test_313/latest.pth.
Evaluation
Test on training human poses:
python run.py --type evaluate --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313
Test on unseen human poses:
python run.py --type evaluate --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313 test_novel_pose True
Then you can get a quantitative evaluation in terms of PSNR and SSIM. (The results may slightly differ from those reported in the paper, because we found that we originally used the incorrect ray sampling code in the Neural Body pipeline. Here we use the updated code.)
Novel view synthesis (rotate camera)
python run.py --type visualize --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313 vis_novel_view True
Shape control (change SMPL shape parameters)
Here we use the list of [pc, val] to specify the principal components of SMPL.
Here, pc represents the number of principal components and val represents the value of principal components.
For example, if you want to set pc1 to 4 and pc2 to -2, run:
python run.py --type visualize --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313 vis_novel_view True shape_control "[[1, 4], [2, -2]]"
Changing clothes
For example, if you want to use the upper body of subject "377" and lower body of subject "390" for rendering, run:
python run.py --type visualize --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313 vis_novel_view True upper_body test_313 lower_body test_377
You may need to adjust the threshold here for a better segmentation.
Training on ZJU-MoCap from scratch
python train_net.py --cfg_file configs/zju_mocap_exp/multi_view_313.yaml exp_name test_313 resume False
Implementation of the dispersed projection
If you want to learn more about the implementation details of the proposed dispersed projection, check here.
Citation
@inproceedings{xu2022sanerf,
author = {Xu, Tianhan and Fujita, Yasuhiro and Matsumoto, Eiichi},
title = {Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis},
booktitle = {CVPR},
year = {2022},
}Acknowledgement
Our implementation is based on the code for the Neural Body (Peng et al., CVPR 2021). We thank the authors for releasing the code, and please also consider citing their work.