Cascaded Human-Object Interaction Recognition

This repository contains the PyTorch implementation for CVPR 2020 Paper "Cascaded Human-Object Interaction Recognition" by Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, Jianbing Shen.

Our proposed method reached the 1st place in ICCV-2019 Person in Context Challenge (PIC19 Challenge), on both Human-Object Interaction in the Wild (HOIW) and Person in Context (PIC) tracks.

Update #1: A new branch (pytorch-1.5.0) is created, with some bugs fixed. The branch will be easier to use. p.s. you will still see a warning on missing keys (e.g., sa.g.conv.bias), and I did not solve it yet but will try to figure it out later.

Update #2: The score of our model (i.e., 66.04%) on HOIW reported in our paper is obtained by an ensemble of multiple models. Here I only provided the best single model that I have, so it is reasonable that the model does not deliver a similar score. I am running the evaluation on HOIW test set, and expect to report my performance for reference this week (hopefully 02.09.2022).

Update #3: With input size (1200, 700), the mAP of the provided weights on HOIW test is around 57%.

Prerequisites

This implementation is based on mmdetection. Please follow INSTALL.md for installation.

The code will work for pytorch=1.5.0, mmdet=1.0rc0+65c1842, and mmcv=0.4.3.

If you encounter problems on *.so files (e.g., undefined symbol in *.so), please try to delete all existing *.so files and rebuild mmdet.

Prepare Dataset

Please find the dataset from the PIC challenge website: http://picdataset.com:8000/challenge/task/download/

For the test-set annotation and evaluation, please refer to https://drive.google.com/drive/folders/15xrIt-biSmE9hEJ2W6lWlUmdDmhatjKt and https://github.com/YueLiao/PIC_HOIW.

I'd like to thank @zgplvyou for sharing me the links.

Please download converted json files from google drive, and put them in the top-most directory.

Download pre-trained weights

Download from Google Drive.

Results on PIC and HOIW datasets are also provided.

Testing

Run testing on the validation set of PIC v2.0

python tools/test_pic.py configs/pic_v2.0/htc_rel_dconv_c3-c5_mstrain_400_1400_x101_64x4d_fpn_20e_train_rel_dcn_semantichead.py pic_latest.pth --json_out det_result.json

Run testing on the validation set of HOIW

python tools/test_hoiw.py configs/hoiw/cascade_rcnn_x101_64x4d_fpn_1x_4gpu_rel.py hoiw_latest.pth --json_out det_result.json --hoiw_out hoiw_result.json

Citation

@article{zhou2021cascaded,
  title={Cascaded parsing of human-object interaction recognition},
  author={Zhou, Tianfei and Qi, Siyuan and Wang, Wenguan and Shen, Jianbing and Zhu, Song-Chun},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={44},
  number={6},
  pages={2827--2840},
  year={2021},
  publisher={IEEE}
}

@inproceedings{zhou2020cascaded,
  title={Cascaded human-object interaction recognition},
  author={Zhou, Tianfei and Wang, Wenguan and Qi, Siyuan and Ling, Haibin and Shen, Jianbing},
  booktitle=CVPR,
  pages={4263--4272},
  year={2020}
}

tfzhou / C-HOI