Gaze following
PyTorch implementation of our ACCV2018 paper:
'Believe It or Not, We Know What You Are Looking at!' [paper] [poster]
Dongze Lian*, Zehao Yu*, Shenghua Gao
(* Equal Contribution)
Prepare training data
GazeFollow dataset is proposed in [1], please download the dataset from http://gazefollow.csail.mit.edu/download.html. Note that the downloaded testing data may have wrong label, so we request test2 provided by author. I do not know whether the author update their testing set. If not, it is better for you to e-mail authors in [1]. For your convenience, we also paste the testing set link here provided by authors in [1] when we request. (Note that the license is in [1])
Download our dataset
OurData is in Onedrive Please download and unzip it
OurData contains data descriped in our paper.
OurData/tools/extract_frame.py
extract frame from clipVideo in 2fps. Different version of ffmpeg may have different results, we provide our extracted images.
OurData/tools/create_video_image_list.py
extract annotation to json.
Testing on gazefollow data
Please download the pretrained model manually and save to model/
cd code
python test_gazefollow.py
Evaluation metrics
cd code
python cal_min_dis.py
python cal_auc.py
Test on our data
cd code
python test_ourdata.py
Training scratch
cd code
python train.py
Inference
simply run python inference.py image_path eye_x eye_y to infer the gaze. Note that eye_x and eye_y is the normalized coordinate (from 0 - 1) for eye position. The script will save the inference result in tmp.png.
cd code
python inference.py ../images/00000003.jpg 0.52 0.14
Reference:
[1] Recasens*, A., Khosla*, A., Vondrick, C., Torralba, A.: Where are they looking? In: Advances in Neural Information Processing Systems (NIPS) (2015).
Citation
If this project is helpful for you, you can cite our paper:
@InProceedings{Lian_2018_ACCV,
author = {Lian, Dongze and Yu, Zehao and Gao, Shenghua},
title = {Believe It or Not, We Know What You Are Looking at!},
booktitle = {ACCV},
year = {2018}
}