This software is provided as a supplementary material for our CVPR'18 paper:
2D/3D Pose Estimation and Action Recognition using Multitask Deep Learning
A demonstration video can be seen here.
During the next few days I will be uploading code and weights corresponding to our CVPR paper, which will be tagged appropriately. Meanwhile, I will still make changes in that code.
Please refer to the installation guide.
The model trained on MPII data reached 91.2% on the test set using multi-crop and horizontal flipping data augmentation, and 89.1% on the validation set, single-crop. To reproduce results on validation, do:
python3 exp/mpii/eval_mpii_singleperson.py output/eval-mpii
The output will be stored in output/eval-mpii/log.txt.
This model was trained using MPII and Human3.6M data. Evaluation on Human3.6M is performed on the validation set. To reproduce our results, do:
python3 exp/h36m/eval_h36m.py output/eval-h36m
The mean per joint position error is 55.1 mm on single crop. Note that some scores on individual activities differ from reported results on the paper. That is because for the paper we computed scores using one frame every 60, instead of using one frame every 64. The average score is the same.
For 2D action recognition, the pose estimation model was trained on mixed data from MPII and PennAction, and the full model for action recognition was trained and fine-tuned on PennAction only. To reproduce our scores, do:
python3 exp/pennaction/eval_penn_ar_pe_merge.py output/eval-penn
For 3D action recognition, the pose estimation model was trained on mixed data from MPII, Human3.6 and NTU, and the full model for action recognition was trained and fine-tuned on NTU only. To reproduce our scores, do:
python3 exp/ntu/eval_ntu_ar_pe_merge.py
Please cite our paper if this software (or any part of it) or weights are useful for you.
@InProceedings{Luvizon_2018_CVPR,
author = {Luvizon, Diogo C. and Picard, David and Tabia, Hedi},
title = {2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}
MIT License