MKnoche / warp3d_reposing

Reposing Humans by Warping 3D Features

Reposing Humans by Warping 3D Features

Markus Knoche, István Sárándi, Bastian Leibe (RWTH Aachen University)

Project | Paper | Video

This repository contains code for our paper “Reposing Humans by Warping 3D Features”. We warp implicitely learned volumetric features with explicit 3D transformations to change the pose of a given person into any desired target pose.

Setup

Using conda:

conda create -n warp3d_reposing
conda activate warp3d_reposing
conda install tensorflow-gpu=1.12
conda install scikit-image
conda install -c menpo opencv3
conda install numpy=1.16

Code Overview

• parameters.py contains global parameters for training, model and dataset
• dataset.py loads pairs of images and their respective poses, estimates the transformations and creates the bodypart-masks
• model.py contains generator and discriminator as well as the warping block
• train.py glues everything together

Datasets

The parameter params['data_dir'] points to the root directory which includes all dataset directories. Each dataset directory contains a directory images with all images organized in subfolders. It further must include a pickle file poses.pkl containing nested dictionaries with the same hierarchy as the image directory. Each image img.png is replaced by a dictionary key img which points to a numpy array with one row per joint and three columns for the coordinates in pixel space. Height and width are between 0 and 256, the depth is centered around 0.

Estimated Poses

poses.pkl for deepfashion, save as <data_dir>/fashion3d/poses.pkl

poses.pkl for iPER, save as <data_dir>/iPER/poses.pkl

Training

First, adapt the paths for the data root directory, the checkpoint directory and the tensorboard directory in parameters.py according to your system structure.

Second, download the pose estimator checkpoint which is used for validation. Extract the three files into pose3d_minimal/checkpoint/.

You now have two options to start a training. You can run train.py with command-line parameters, for example

python3 train.py name feat_weight_5 feature_loss_weight 5.

Make sure to always include a unique name, because a new directory with this name will be created in the checkpoint directory and the tensorboard directory.

To keep track of different parameter combinations, you can also define a JOB_ID in parameters.py and then for example use

python3 train.py JOB_ID 5.

For each ablation model of our paper, a JOB_ID is already defined.

Pretrained Models

Extract the checkpoints into <check_dir>/<model_name>/.

iPER

name	3D warping	3D target pose	checkpoint
iPER-3d_w-3d_p	✔️	✔️	iPER-3d_w-3d_p.zip
iPER-3d_w-2d_p	✔️	➖	iPER-3d_w-2d_p.zip
iPER-2d_w-3d_p	➖	✔️	iPER-2d_w-3d_p.zip
iPER-2d_w-2d_p	➖	➖	iPER-2d_w-2d_p.zip

deepfashion

name	3D warping	3D target pose	checkpoint
fash-3d_w-3d_p	✔️	✔️	fash-3d_w-3d_p.zip
fash-3d_w-2d_p	✔️	➖	fash-3d_w-2d_p.zip
fash-2d_w-3d_p	➖	✔️	fash-2d_w-3d_p.zip
fash-2d_w-2d_p	➖	➖	fash-2d_w-2d_p.zip

BibTeX

@inproceedings{Knoche20CVPRW,
 author = {Markus Knoche and Istv\'an S\'ar\'andi and Bastian Leibe},
 title = {Reposing Humans by Warping 3{D} Features},
 booktitle = {CVPR Workshop on Towards Human-Centric Image/Video Synthesis},
 year = {2020}
}