Transformer-based Unified Recognition of Two Hands Manipulating Objects

Official code of Transformer-based Unified Recognition of Two Hands Manipulating Objects

Hoseong Cho, Chanwoo Kim, Jihyeon Kim, Seongyeong Lee, Elkhan Ismayilzada, Seungryul Baek

Input	Pose	Contact

Installation

git clone https://github.com/chohoseong/H2OTR.git

conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install -r requirements.txt
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d

Compiling CUDA operators

cd ./models/ops
sh ./make.sh

.H2OTR
├── AIK
│        ├── AIK_config.py
│        └── AIK_torch.py
├── config
│        ├── H2O 
│        └── FPHA
├── mano
│        ├── models
│        └── ...
├── manopth
│        ├── manolayer.py
│        └── ...
├── models
│        ├── H2OTR.py
│        └── ...
├── README.md
├── demo.py
├── requirements.txt
...

Data Preparation

Download FPHA, H2O datasets.

.H2OTR
└── data
    ├── FPHA
    │   ├── Object_6D_pose_annotation_v1
    │   ├── Video_files
    │   ├── data_split_action_recognition.txt
    │   └── ...
    └── H2O
        ├── action_labels
        ├── subject1
        ├── subject2
        └── ...

Pretrained weight

You can access the pretrained weight file for the H2O and FPHA datasets by following the link

Inference

python demo.py \
        --dataset_file H2O \
        --pretrained_model weights/H2O.pth \
        --vid_id 9

Citation

@inproceedings{cho2023transformer,
  title={Transformer-Based Unified Recognition of Two Hands Manipulating Objects},
  author={Cho, Hoseong and Kim, Chanwoo and Kim, Jihyeon and Lee, Seongyeong and Ismayilzada, Elkhan and Baek, Seungryul},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={4769--4778},
  year={2023}
}

chohoseong / H2OTR