Triplet-Representation-of-human-Body

This repo is for paper "TRB: A Novel Triplet Representation for 2D Human Human Body Understanding".. The repo contains contour keypoints annotations of MPII dataset used in this paper.

TRB is a novel 2D Human Body Representation we designed, which contains both 2D human skeleton and contour, in the form of skeleton keypoints and contour keypoints. We use points defined in current 2D pose estimation dataset as skeleton keypoints. For each skeleton keypoint, two contour points located on human boundary are defined as contour keypoints, the three keypoints form a triplet.

In TRB definition, contour keypoints are divided into medial points and lateral points, corresponding to their relative position. In the demonstration below, red, blue, green points denote skeleton points, medial contour points, lateral contour points respectively.

We try to use TRB as supervision for skeleton keypoint estimation task. When combined with TRB-Net, we get significant improvement over the baseline. On MPII and LSP, we use paper A Cascaded Inception of Inception Network with AttentionModulated Feature Fusion for Human Pose Estimation as our baseline. On COCO, we use 2-stack hourglass as our baseline.
On LSP Dataset (threshold PCK@0.2):

LSP Test Head Shoulder Elbow Wrist HIp Knee Ankle Mean

Baseline 98.1 94.0 91.0 89.0 93.4 95.2 94.4 93.6

with TRB 98.5 95.3 92.6 90.6 93.8 95.8 95.5 94.5
On MPII Dataset (threshold PCKh@0.5):

MPII Test Head Shoulder Elbow Wrist HIp Knee Ankle Mean

Baseline 98.4 96.4 92.0 87.8 90.7 88.3 85.3 91.6

with TRB 98.5 96.6 92.6 88.3 91.6 89.2 86.5 92.2
On COCO Dataset (Half human instances are annotated with TRB, using detected bbox)

COCO Val AP AP⁵⁰ AP⁷⁵ AP^M AP^L

Baseline 71.9 88.9 78.4 68.7 77.2

with TRB 73.6 89.6 80.3 70.1 79.0

LSP Test	Head	Shoulder	Elbow	Wrist	HIp	Knee	Ankle	Mean
Baseline	98.1	94.0	91.0	89.0	93.4	95.2	94.4	93.6
with TRB	98.5	95.3	92.6	90.6	93.8	95.8	95.5	94.5

MPII Test	Head	Shoulder	Elbow	Wrist	HIp	Knee	Ankle	Mean
Baseline	98.4	96.4	92.0	87.8	90.7	88.3	85.3	91.6
with TRB	98.5	96.6	92.6	88.3	91.6	89.2	86.5	92.2

COCO Val	AP	AP⁵⁰	AP⁷⁵	AP^M	AP^L
Baseline	71.9	88.9	78.4	68.7	77.2
with TRB	73.6	89.6	80.3	70.1	79.0

TRB can be used as guidance for pose and shape guided human image generation. We show some examples below, in which we change contour points location to manipulate generated human shape.

Leg Editing Shoulder Editing Torso Editing

File 'MPII_contour_points.json' contains contour keypoints annotation for MPII train-val. It can be aligned with MPII dataset using image name and human center, to construct the MPII-TRB dataset.

kennymckormick / Triplet-Representation-of-human-Body