chaitanya100100 / TailorNet

Code for our CVPR 2020 (ORAL) paper - TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style.

TailorNet

This repository contains training and inference code for the following paper:

TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style  
Chaitanya Patel*, Zhouyingcheng Liao*, Gerard Pons-Moll  
CVPR 2020 (ORAL)  

[ArXiv] [Project Website] [Dataset Repo] [Oral Presentation] [Results Video]

Citation

Cite us if you use our model, code or data:

@inproceedings{patel20tailornet,
        title = {TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style},
        author = {Patel, Chaitanya and Liao, Zhouyingcheng and Pons-Moll, Gerard},
        booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)},
        month = {jun},
        organization = {{IEEE}},
        year = {2020},
    }

Updates

• [26-12-2020] Female skirt weights added.
• [11-11-2020] Female and male short-pant weights added.
• [02-08-2020] Female and male pant weights added.
• [19-07-2020] Male shirt weights added.
• [12-07-2020] Female shirt weights added.
• [28-06-2020] Female t-shirt weights added.
• [25-06-2020] Minor bug fixes and male t-shirt weights added.
• [17-06-2020] Inference script and female old-t-shirt weights added.

Requirements

python3
pytorch
scipy
chumpy
psbody.mesh

Code works with psbody.mesh v0.4 , pytorch >= v1.0 , chumpy v0.7 and scipy v1.3 .

How to Run

• Download and prepare SMPL model and TailorNet data from dataset repository.
• Set DATA_DIR and SMPL paths in global_var.py file accordingly.
• Download trained model weights in a directory and set its path to MODEL_WEIGHTS_PATH variable in global_var.py.
  • old-t-shirt_female_weights (4.1 GB)
  • t-shirt_male_weights (2.0 GB)
  • t-shirt_female_weights (2.0 GB)
  • shirt_female_weights (2.5 GB)
  • shirt_male_weights (2.5 GB)
  • This link has all the weights listed above as well as the following:
    • pant_female_weights
    • pant_male_weights
    • short-pant_female_weights
    • short-pant_male_weights
    • skirt_female_weights
• Set output path in run_tailornet.py and run it to predict garments on some random inputs. You can play with different inputs. You can also run inference on motion sequence data.
• To visualize predicted garment using blender, run python run_tailornet.py render. (Blender 2.79 needs to be installed.)

TailorNet Per-vertex Error in mm on Test Set

... evaluated using evaluate function in utils/eval.py.

garment_class	gender	TailorNet Baseline	TailorNet Mixture Model
old-t-shirt	female	11.1	10.7
t-shirt	female	12.6	12.3
t-shirt	male	11.4	11.2
shirt	female	14.2	14.1
shirt	male	12.7	12.5
pant	female	4.7	4.8
pant	male	8.1	8.1
short-pant	female	6.8	6.6
short-pant	male	7.0	7.0
skirt	female	7.7	7.8

Training TailorNet yourself

• Set global variables in global_var.py, especially LOG_DIR where training logs will be stored.
• Set config variables like gender and garment class in trainer/base_trainer.py (or pass them via command line) and run python trainer/base_trainer.py to train TailorNet MLP baseline.
• Similarly, run python trainer/lf_trainer.py to train low frequency predictor and trainer/ss2g_trainer.py to train shape-style-to-garment(in canonical pose) model.
• Run python trainer/hf_trainer.py --shape_style <shape1>_<style1> <shape2>_<style2> ... to train pivot high frequency predictors for pivots <shape1>_<style1>, <shape2>_<style2>, and so on. See DATA_DIR/<garment_class>_<gender>/pivots.txt to know available pivots.
• Use models.tailornet_model.TailorNetModel with appropriate logdir arguments to do prediction.

Inference Time

In the paper, we report inference time to be 1-2 ms per frame(depending upon garment) which is averaged inference time over the batch of 21 samples(20-40 ms per batch). Apologies for the ambiguity. Running each sample separately takes almost same time as batch - around 20 ms per frame for all garments. However, note that TailorNet has 21 independent MLPs, so we believe that faster inference time is possible if MLPs are configured to run in parallel on GPU cores.

Misc

• See ./models/skirt_model.md for the explanation of skirt garment model.
• Thanks to Bharat for many fruitful discussions and for smpl_lib library taken from his MultiGarmentNet repo's lib folder.
• Thanks to Garvita for helping out during the onerous procedure of data generation.

For any doubt or concert about the code, raise an issue on this repository.