junkunyuan / HAP

[NeurIPS 2023] HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception

HAP

📚 Contents

• HAP
  • 📚 Contents
  • 📋 Introduction
  • 📂 Datasets
  • 🛠️ Environment
  • 🚀 Get Started
  • 🏆 Results
  • 💗 Acknowledgement
  • ✅ Citation
  • 🤝 Contribute & Contact

📋 Introduction

This repository contains the implementation code for paper:

HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception

Advances in Neural Information Processing Systems (NeurIPS) 2023

[arXiv]   [project page]

HAP is the first masked image modeling framework for human-centric pre-training. It leverages body structure-aware training to learn general human visual representations. It achieves SOTA performance across several human-related benchmarks.

📂 Datasets

Pre-Training Data

We use LUPerson for pre-training. To make the pre-training more efficient, we only use half of the dataset with a list named "CFS_list.pkl" from TransReID-SSL. To extract the keypoint information of data, which is the masking guidance during pre-training, ViTPose is used to perform inference on LUPerson. You can download our pose dataset here.

Put the dataset directories outside the HAP project:

root
├── HAP
├── LUPerson-data  # LUPerson data
│   ├── xxx.jpg
│   └── ...
└── LUPerson-pose  # LUPerson with pose keypoints
    ├── xxx.npy
    └── ...

🛠️ Environment

Conda is recommended for configuring the environment:

conda env create -f env-hap.yaml && conda activate env_hap

🚀 Get Started

The default setting for pre-training is 400 epochs with total batch-size of 4096.

It may need 32 GPUs with memory larger than 32GB, such as NVIDIA V100, for pre-training.

# -------------------- Pre-Training HAP on LUPerson --------------------
cd HAP/

MODEL=pose_mae_vit_base_patch16

# Download official MAE model pre-trained on ImageNet and move it here
CKPT=mae_pretrain_vit_base.pth

# Download cfs list and move it here
CFS_PATH=cfs_list.pkl

OMP_NUM_THREADS=1 python -m torch.distributed.launch \
    --nnodes=${NNODES} \
    --node_rank=${RANK} \
    --master_addr=${ADDRESS} \
    --master_port=${PRETRAIN_PORT} \
    --nproc_per_node=${NPROC_PER_NODE} \
    main_pretrain.py \
    --dataset LUPersonPose \
    --data_path ../LUPerson-data \
    --pose_path ../LUPerson-pose \
    --sample_split_source ${CFS_PATH} \
    --batch_size 256 \
    --model ${MODEL} \
    --resume ${CKPT} \
    --ckpt_pos_embed 14 14 \
    --mask_ratio 0.5 \
    --align 0.05 \
    --epochs 400 \
    --blr 1.5e-4 \
    --ckpt_overwrite \
    --seed 0 \
    --tag default

🏆 Results

We evaluate HAP for the following downstream tasks. Click them to find implementation instructions.

• Person ReID
• Text-to-Image ReID
• 2D Pose estimation
• 3D Pose and Shape Estimation
• Pedestrian Attribute Recognition

You can download the checkpoint of the pre-trained HAP model here. The results are given below.

task	dataset	resolution	structure	result
Person ReID	MSMT17	(256, 128)	ViT	76.4 (mAP)
Person ReID	MSMT17	(384, 128)	ViT	76.8 (mAP)
Person ReID	MSMT17	(256, 128)	ViT-lem	78.0 (mAP)
Person ReID	MSMT17	(384, 128)	ViT-lem	78.1 (mAP)
Person ReID	Market-1501	(256, 128)	ViT	91.7 (mAP)
Person ReID	Market-1501	(384, 128)	ViT	91.9 (mAP)
Person ReID	Market-1501	(256, 128)	ViT-lem	93.8 (mAP)
Person ReID	Market-1501	(384, 128)	ViT-lem	93.9 (mAP)

task	dataset	resolution	training	result
2D Pose Estimation	MPII	(256, 192)	single-dataset	91.8 (PCKh)
2D Pose Estimation	MPII	(384, 288)	single-dataset	92.6 (PCKh)
2D Pose Estimation	MPII	(256, 192)	multi-dataset	93.4 (PCKh)
2D Pose Estimation	MPII	(384, 288)	multi-dataset	93.6 (PCKh)
2D Pose Estimation	COCO	(256, 192)	single-dataset	75.9 (AP)
2D Pose Estimation	COCO	(384, 288)	single-dataset	77.2 (AP)
2D Pose Estimation	COCO	(256, 192)	multi-dataset	77.0 (AP)
2D Pose Estimation	COCO	(384, 288)	multi-dataset	78.2 (AP)
2D Pose Estimation	AIC	(256, 192)	single-dataset	31.5 (AP)
2D Pose Estimation	AIC	(384, 288)	single-dataset	37.7 (AP)
2D Pose Estimation	AIC	(256, 192)	multi-dataset	32.2 (AP)
2D Pose Estimation	AIC	(384, 288)	multi-dataset	38.1 (AP)

task	dataset	result
Pedestrian Attribute Recognition	PA-100K	86.54 (mA)
Pedestrian Attribute Recognition	RAP	82.91 (mA)
Pedestrian Attribute Recognition	PETA	88.36 (mA)

task	dataset	result
Text-to-Image Person ReID	CUHK-PEDES	68.05 (Rank-1)
Text-to-Image Person ReID	ICFG-PEDES	61.80 (Rank-1)
Text-to-Image Person ReID	RSTPReid	49.35 (Rank-1)

task	dataset	result
3D Pose Estimation	3DPW	90.1 (MPJPE), 56.0 (PA-MPJPE), 106.3 (MPVPE)

💗 Acknowledgement

We acknowledge the following open source projects.

• Model:   MAE   MALE   BEiT  
• Dataset:   LUPerson   TransReID-SSL   ViTPose  
• Downstream evaluation:   MALE   ViTPose   mmcv   mmpose   Rethinking_of_PAR   LGUR   3DCrowdNet
• Else:   Swin  

✅ Citation

@article{yuan2023hap,
  title={HAP: Structure-Aware Masked Image Modeling for Human-Centric Perception},
  author={Yuan, Junkun and Zhang, Xinyu and Zhou, Hao and Wang, Jian and Qiu, Zhongwei and Shao, Zhiyin and Zhang, Shaofeng and Long, Sifan and Kuang, Kun and Yao, Kun and others},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2023}
}

🤝 Contribute & Contact

Feel free to star and contribute to our repository.

If you have any questions or advice, contact us through GitHub issues or email ([e-mail]).