Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations
Atsuhiro Noguchi, Xiao Sun, Stephen Lin, Tatsuya Harada
git clone --recursive git@github.com:nogu-atsu/ENARF-GAN.git
cd ENARF-GAN
conda create -n enarfgan python=3.9
conda activate enarfgan
pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
cd cuda_extention
python setup.py install
# For dataset preprocessing
cd ../
git clone git@github.com:google/aistplusplus_api.git
cd aistplusplus_api
pip install -r requirements.txt
python setup.py install
We have only tested the code on NVIDIA A100, A6000, and RTX3080Ti GPUs.
If you get RuntimeError: Ninja is required to load C++ extension, this may be helpful.
You only need to generate sample data for the demo.
Dictionary of all data is stored in a single pickle file.
{
"img": numpy array of all images. each image is compressed by blosc. [N],
"camera_intrinsic": camera intrinsic matrix [N, 3, 3],
"camera_rotation": camera rotation matrix (optional) [N, 3, 3],
"camera_translation": camera translation matrix (optional)[N, 3, 1],
"smpl_pose": pose of SMPL. pose is in world coordinate if camera rotation and translation are provided, otherwise in camera coordinate.[N, 24, 4, 4],
"frame_id": frame index of video (optional) [N]
}- Download the SMPL models
following EasyMocap installation. You only need
to download smplx models.
smpl_data ├── J_regressor_body25.npy ├── J_regressor_body25_smplh.txt ├── J_regressor_body25_smplx.txt ├── J_regressor_mano_LEFT.txt ├── J_regressor_mano_RIGHT.txt ├── SMPLX_FEMALE.pkl ├── SMPLX_MALE.pkl └── SMPLX_NEUTRAL.pkl - Run
cd data_preprocess python prepare_canonical_pose.py
- Download the DeepCap dataset used for training Neural Actor from here (S1_marc.zip, S2_lan.zip)
- Unzip them and
transform.zipinside them<path_to_data> ├── lan │ ├── intrinsic │ └── ... └── marc ├── intrinsic └── ... - Sample data generation
cd data_preprocess python NeuralActor/prepare_sample_data.py --data_path <path_to_NeuralActor> --person_name lan python NeuralActor/prepare_sample_data.py --data_path <path_to_NeuralActor> --person_name marc - Training data generation
cd data_preprocess python NeuralActor/preprocess.py --data_path <path_to_NeuralActor>
- Download the ZJU MOCAP dataset
used for training AnimatableNeRF
<path_to_zju> ├── CoreView_313 ├── CoreView_315 └── CoreView_386 - Sample data generation
cd data_preprocess python ZJU/prepare_sample_data.py --data_path <path_to_zju> --person_id 313 - Training data generation
cd data_preprocess python ZJU/preprocess.py --data_path <path_to_zju>
- Download the SURREAL dataset as
<path_to_surreal> ├── test ├── train └── val - Sample data generation
cd data_preprocess python surreal/prepare_sample_data.py --data_path <path_to_surreal> - Training data generation
cd data_preprocess python surreal/preprocess.py --data_path
- Download the AIST++ dataset as
<path_to_aist++> ├── gLO_sBM_c07_d15_mLO5_ch03.mp4 └── ... <path_to_annotation> ├── camearas ├── ingore_list.txt ├── keypoints2d ├── keypoints3d ├── motions └── splits - Sample data generation
cd data_preprocess python AIST/prepare_sample_data.py --data_path <path_to_aist++> --annotation_path <path_to_annotatin> - Training data generation
cd data_preprocess python AIST/preprocess.py --data_path <path_to_aist++> --annotation_path <path_to_annotatin>
Please run sample data generation before running the demo.
- Download the pretrained models from here
data └── result ├── DSO │ ├── NeuralActor │ │ ├── lan_denarf │ │ │ └── snapshot_latest.pth │ │ └── ... │ └── ZJU │ ├── 313_denarf │ │ └── snapshot_latest.pth │ └── ... └── GAN ├── AIST │ └── enarfgan │ └── snapshot_latest.pth └── SURREAL └── enarfgan └── snapshot_latest.pth
- Example command for running the DSO model
python DSO_demo.py --config configs/DSO_demo/NeuralActor/lan_denarf.yml - Synthesized images are saved in
data/result/DSO/NeuralActor/lan_denarf/samples
- Example command for running the GAN model
python ENARF_GAN_demo.py --config configs/enarfgan_demo/AIST/enarfgan.yml - Synthesized images are saved in
data/result/GAN/AIST/enarfgan/samples - If you get Out of Memory error, please try multiple times or reduce
ray_batchsizeinlibraries/NARF/mesh_rendering.py
We tested training on a single A100 GPU.
- Example command for training the DSO model
python train_DSO.py --config configs/DSO_train/ZJU/313_denarf.yml --default_config configs/DSO_train/default.yml - Example command for training the GAN model
python train_ENARF_GAN.py --config configs/enarfgan_train/AIST/config.yml --default_config configs/enarfgan_train/default.yml - Results are saved in
data/result/DSO/ZJU/313_denarfanddata/result/GAN/AIST/example - If there is not enough memory, try reducing bs (batchsize) or increasing n_accum_step in config.
python train_DSO.py --validation --config configs/DSO_train/NeuralActor/lan_denarf.yml --num_workers 2 --resume_latest
Please install mmpose before running compute_PCK.py
python evaluation/compute_depth.py --config configs/enarfgan_train/SURREAL/config.yml --num_workers 2 --iteration -1 --truncation 0.4
python evaluation/compute_PCK.py --config configs/enarfgan_train/SURREAL/config.yml --num_workers 2 --iteration -1 --truncation 0.4
python evaluation/compute_fid.py --config configs/enarfgan_train/SURREAL/config.yml --num_workers 2 --iteration -1
If you find this work useful for your research, please cite:
@inproceedings{noguchi2022unsupervised,
author = {Noguchi, Atsuhiro and Sun, Xiao and Lin, Stephen and Harada, Tatsuya},
title = {Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations},
booktitle = {European Conference on Computer Vision},
year = {2022},
}