R(2+1)D and Mixed-Convolutions for Action Recognition
[project page] [paper]
If you find this work helpful for your research, please cite our following paper:
D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri. A Closer Look at Spatiotemporal Convolutions for Action Recognition. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
@inproceedings{r2plus1d_cvpr18,
title = {A Closer Look at Spatiotemporal Convolutions for Action Recognition},
author = {Du Tran and Heng Wang and Lorenzo Torresani and Jamie Ray and Yann LeCun and
Manohar Paluri},
booktitle = {CVPR},
year = 2018
}
If you have any question or feedback about the code, please contact: trandu@fb.com, hengwang@fb.com.
Requirements
R2Plus1D requires the following dependencies:
- OpenCV (tested on 3.4.1) and ffmpeg.
- Caffe2 and its dependencies.
- You will need to build from source and install with
USE_OPENCV=1 USE_FFMPEG=1 USE_LMDB=1 python setup.py installfor OpenCV, ffmpeg, and lmdb support.
- You will need to build from source and install with
- And lmdb, python-lmdb, and pandas.
Installation
- You need to install ffmpeg, OpenCV, and caffe2. Caffe2 source build instructions can be found here but make sure you install with
USE_OPENCV=1 USE_FFMPEG=1 USE_LMDB=1 python setup.py install. You also need to install lmdb, python-lmdb, and pandas.
Tutorials
We provide some basic tutorials for you to get familar with the code and tools.
- Installation Guide
- Training Kinetics from scratch
- Finetuning R(2+1)D on HMDB51
- Dense prediction
- Feature extraction
- Download and evaluate pre-trained models
License
R2Plus1D is Apache 2.0 licensed, as found in the LICENSE file.
Acknowledgements
The authors would like to thank Ahmed Taei, Aarti Basant, Aapo Kyrola, and the Facebook Caffe2 team for their help in implementing ND-convolution, in optimizing video I/O, and in providing support for distributed training. We are grateful to Joao Carreira for sharing I3D results on the Kinetics validation set.