PyTorch-JAANet

This repository is the PyTorch implementation of JAA-Net, as well as its extended journal version. "*v1.py" is for the ECCV version, and "*v2.py" is for the IJCV version. The trained models can be downloaded here. The original Caffe implementation can be found here

Getting Started

Installation

This code was tested with PyTorch 1.1.0 and Python 3.5
Clone this repo:

git clone https://github.com/ZhiwenShao/PyTorch-JAANet
cd PyTorch-JAANet

Datasets

BP4D and DISFA

Put these datasets into the folder "dataset" following the paths shown in the list files of the folder "data/list". You can refer to the example images for BP4D and DISFA

Preprocessing

Put the landmark annotation files into the folder "dataset". Two example files "BP4D_combine_1_2_land.txt" and "DISFA_combine_1_2_66land.txt" are also provided
- For DISFA dataset, we need to select the annotations of 49 landmarks from original 66 landmarks:
```
cd dataset
python read_disfa_49land.py
```
Conduct similarity transformation for face images:
```
cd dataset
python face_transform.py
```
Compute the inter-ocular distance of each face image:
```
cd dataset
python write_biocular.py
```
Compute the weight of the loss of each AU for the training set:
- The AU annoatation files should be in the folder "data/list"
```
cd dataset
python write_AU_weight.py
```

Training

Train on BP4D with the first two folds for training and the third fold for testing:

python train_JAAv1.py --run_name='JAAv1' --gpu_id=0 --train_batch_size=16 --eval_batch_size=28 --train_path_prefix='data/list/BP4D_combine_1_2' --test_path_prefix='data/list/BP4D_part3' --au_num=12

Train on DISFA with the first two folds for training and the third fold for testing, using the the well-trained BP4D model for initialization:

python train_JAAv1_disfa.py --run_name='JAAv1_DISFA' --gpu_id=0 --train_batch_size=16 --eval_batch_size=32 --train_path_prefix='data/list/DISFA_combine_1_2' --test_path_prefix='data/list/DISFA_part3' --au_num=8 --pretrain_path='JAAv1_combine_1_3' --pretrain_epoch=5

Testing

Test the models saved in different epochs:

python test_JAAv1.py --run_name='JAAv1' --gpu_id=0 --start_epoch=1 --n_epochs=12 --eval_batch_size=28 --test_path_prefix='data/list/BP4D_part3' --au_num=12

Visualize attention maps

python test_JAAv1.py --run_name='JAAv1' --gpu_id=0 --pred_AU=False --vis_attention=True --start_epoch=5 --n_epochs=5 --test_path_prefix='data/list/BP4D_part3' --au_num=12

Supplement

The PyTorch implementation for the ECCV version conducts two minor revisions to make the proposed method more general:
- The redundant cropping of attention maps is removed
- The assembling of local feature maps uses element-wise average instead of element-wise sum
The differences in the extended journal version are detailed here

Citation

If you use this code for your research, please cite our papers

@inproceedings{shao2018deep,
  title={Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment},
  author={Shao, Zhiwen and Liu, Zhilei and Cai, Jianfei and Ma, Lizhuang},
  booktitle={European Conference on Computer Vision},
  year={2018},
  pages={725--740},
  organization={Springer}
}
@article{shao2020jaa,
  title={J{\^A}A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention},
  author={Shao, Zhiwen and Liu, Zhilei and Cai, Jianfei and Ma, Lizhuang},
  journal={International Journal of Computer Vision},
  year={2020},
  publisher={Springer}
}

jingyang2017 / PyTorch-JAANet