Team Rayee, ranks 1st place in reconstructing partial textured objects (track 2), and 2nd overall in the SHApe Recovery from Partial textured 3D scans (SHARP) Challenge @ CVPR 2022.
This repository contains the implementation of the following techinical report:
3D Textured Shape Recovery with Learned Geometric Priors
Lei Li*, Zhizheng Liu*, Weining Ren*, Liudi Yang*, Fangjinhua Wang, Marc Pollefeys and Songyou Peng (* denotes equal contribution).
If you find our code or report useful, please consider citing
@article{Li2022SHARP,
author = {Lei Li, Zhizheng Liu, Weining Ren, Liudi Yang, Fangjinhua Wang, Marc Pollefeys and Songyou Peng},
title = {3D Textured Shape Recovery with Learned Geometric Priors},
journal= {ArXiv},
year = {2022},
volume = {abs/2209.03254}
}Our project is built upon the following awesome projects:
- Implict Feature Networks by [Chibane et. al. CVPR'20]
- Convolutional Occupancy Networks by [Peng et. al. ECCV'20]
- SMPL model by [Loper et. al. SIGGRAPH Asia'15] and its PyTorch extension
Please cite them if you use our code.
Please clone the repository and navigate into it in your terminal, its location is assumed for all subsequent commands. We use pytorch 1.8.1 with cuda 10.2 to build the environment, please install the following dependencies
pip install -r requirements.txt
Install the needed libraries with:
cd src/data_processing/libmesh/
python setup.py build_ext --inplace
cd src/data_processing/libvoxelize/
python setup.py build_ext --inplace
Please participate in the Sharp challenge to gain access of the dataset and the preprocessing tools. Download and unzip the data of Challange 1 into dataset/SHARP2022/ such that the data is stored as dataset/SHARP2022/challenge1/{track1,track2}/{test,train}/File_ID/Files, we show the data preparation procedures for track 1 and track 2 data can be prepared in a similar manner.
Next, folow their official data preprocessing protocal, we create 4 different, random partial scans for each complete human scan with
python -m sharp_challenge1 do_crop_dir dataset/SHARP2022/challenge1-track1 dataset/SHARP2022/challenge1-track1 --nViews 4
and convert everything to .obj-Format.
python src/data_processing/convert_to_obj.py
We sample points near the ground truth complete surface, extract their occupancy and save it as training data with:
python src/data_processing/boundary_sampling.py config/SHARP2022_geometry/track1_geometry.yaml --sigma 0.015
python src/data_processing/boundary_sampling.py config/SHARP2022_geometry/track1_geometry.yaml --sigma 0.2
Next, we create the input for the geometry network(an incomplete voxelized shape) with
python src/data_processing/voxelized_pointcloud_sampling.py config/SHARP2022_geometry/track1_geometry.yaml
python src/data_processing/generate_pose3d.py config/SHARP2022_geometry/track1_geometry.yaml
To sample points on the full colored surface, extract the corresponding RGB color and save it as training data with
python src/data_processing/sample_RGB_GT.py config/SHARP2022_texture/track1_texture.yaml
Next, we create the input for the texture network (a colored, incomplete shape and a complete but uncolored shape) with
python src/data_processing/sample_voxelized_colored_pointcloud.py config/SHARP2022_texture/track1_texture.yaml
To generate SMPL models and meshes of the predicted poses for shape fusion, first create a folder smpl_models and download the SMPL base models to that folder, and move all the predicted poses to dataset/SHARP2022/challenge1/track1/train_smpl, then run
python src/data_processing/smpl_tools/pose2smpl.py SHARP
python src/data_processing/smpl_tools/smpl2obj.py SHARP
Finally, a data split into training, test and validation is created with
python src/data_processing/create_split.py config/SHARP2022_geometry/track1_geometry.yaml
A model is trained to predict the complete shape and color of a partial untextured scan. First, train the task specic networks (pose prediction for task1 and bbox prediciton for task2) as
python -m scripts.train config/SHARP2022_geometry/track1_geometry_pose.yaml
python -m scripts.train config/SHARP2022_geometry/track2_geometry_bbox.yaml
Then, run
python -m scripts.train config/SHARP2022_geometry/track1_geometry.yaml
to train the shape completion network, and use
python -m scripts.train config/SHARP2022_texture/track1_texture.yaml
to train the color prediction network.
To test the performance of a checkpoint such as the chamfer distance, run
python -m scripts.test config/SHARP2022_geometry/track1_geometry.yaml --weights {path_to_ckpt}
To generate predictions on the test dataset for result submission, use
python -m scripts.generate config/SHARP2022_geometry/track1_geometry.yaml --weights {path_to_ckpt}
python -m scripts.generate config/SHARP2022_texture/track1_texture.yaml --weights {path_to_ckpt}
Please see the specific config files for detailed model and training configurations.
For questions and comments regarding the code please contact Zhizheng Liu by email.