CLIFF [ECCV 2022 Oral]

(This testing video is from the 3DPW testset, and processed frame by frame without temporal smoothing.)

Introduction

This repo contains the demo code, and the pseudo-GT SMPL parameters used in the training for the paper:

CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation.

Installation

conda create -n cliff python=3.10
pip install -r requirements.txt

Download the SMPL models for rendering the reconstructed meshes
Download pytorch-yolo-v3, unzip it under the lib directory, and change the folder name to "pytorch_yolo_v3_master"
Download the pretrained checkpoints and some testing samples to run the demo [Google Drive]
Download the CLIFF pseudo-GT for the in-the-wild datasets [Google Drive]

Finally put these data following the directory structure as below:

${ROOT}
|-- cliffGT_v1
    |-- coco2014part_cliffGT.npz
    |-- mpii_cliffGT.npz
|-- data
    |-- ckpt
        |-- hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pt
        |-- res50-PA45.7_MJE72.0_MVE85.3_3dpw.pt
        |-- yolov3.weights
        |-- hr48-PA53.7_MJE91.4_MVE110.0_agora_val.pt
    |-- smpl
        |-- SMPL_FEMALE.pkl
        |-- SMPL_MALE.pkl
        |-- SMPL_NEUTRAL.pkl
|-- lib
    |-- pytorch_yolo_v3_master

Demo

sh scripts/run_demo.sh

The results will be saved in the same directory as the input, including the detected bbox, the front-view and side-view rendering of the reconstructed meshes, and the npz file containing all the estimation information.

One can change the demo options in the script. Please see the option description in the bottom lines of demo.py.

Pseudo-GT

Run the script to visualize the pseudoGT, press the <- and -> button to the next sample, and esc to exit.

sh scripts/run_cliffGT_visualization.sh

Data format:

'imgname':     # image name, e.g., images/015601864.jpg, train2014/COCO_train2014_000000044474.jpg 
'center':      # bbox center, [x, y]
'scale':       # bbox scale, bbox_size_in_pixel / 200.
'part':        # 2D keypoint annotation, shape (24, 3), [x, y, conf], see common/skeleton_drawer.py for the order
'annot_id':    # annotation ID, only available for the COCO dataset
'pose':        # SMPL pose parameters in axis-angle, shape (72,)
'shape':       # SMPL shape parameters, shape (10,)
'has_smpl':    # whether the smpl parameters are available (true for all samples)
'global_t':    # Pelvis translation in the camera coordinate system w.r.t the original full-frame image
'focal_l':     # estimated focal length for the original image, np.sqrt(img_w ** 2 + img_h ** 2)
'S':           # 3D joints with Pelvis aligned at (0, 0, 0), shape (24, 4), [x, y, z, conf], same order as 'part'

Issues

When submitting an issue, please add "[CLIFF]" before the title, e.g., [CLIFF] your issue title. This is for clarifying which subproject the issue points to.

Citing

@Inproceedings{li2022cliff,
  Title     = {CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation},
  Author    = {Li, Zhihao and Liu, Jianzhuang and Zhang, Zhensong and Xu, Songcen and Yan, Youliang},
  Booktitle = {ECCV},
  Year      = {2022}
}

swsong89 / cliff