VMZ is a Caffe2 codebase for video modeling developed by the Computer Vision team at Facebook AI. The aim of this codebase is to help other researchers and industry practitioners:
- reproduce some of our research results and
- leverage our very strong pre-trained models.
Currently, this codebase supports the following models:
- R(2+1)D, MCx models [1].
- CSN models [2].
- R(2+1)D and CSN models pre-trained on large-scale (65 million!) weakly-supervised public Instagram videos (IG-65M) [3].
We provide our latest video models including R(2+1)D, ir-CSN, ip-CSN (all with 152 layers) which are pre-trained on Sports-1M or IG-65M, then fine-tuned on Kinetics-400. Both pre-trained and fine-tuned models are provided in the table below. We hope these models will serve as valuable baselines and feature extractors for the related video modeling tasks such as action detection, video captioning, and video Q&A. More models, e.g. shallower or with shorter clip input are also provided in the model zoo.
| Input size | Pretrained dataset | Pretrained model | Video@1 Kinetics | Video@5 Kinetics | Finetuned model | GFLOPs | params(M) |
|---|---|---|---|---|---|---|---|
| 32x112x112 | Sports1M | link | 79.5 | 94.0 | link | 329.1 | 118.0 |
| 32x112x112 | IG-65M | link | 81.6 | 95.3 | link | 329.1 | 118.0 |
| Input size | Pretrained dataset | Pretrained model | Video@1 Kinetics | Video@5 Kinetics | Finetuned model | GFLOPS | params(M) |
|---|---|---|---|---|---|---|---|
| 32x224x224 | Sports1M | link | 78.2 | 93.0 | link | 96.7 | 29.6 |
| 32x224x224 | IG-65M | link | 82.6 | 95.3 | link | 96.7 | 29.6 |
| Input size | Pretrained dataset | Pretrained model | Video@1 Kinetics | Video@5 Kinetics | Finetuned model | GFLOPS | params(M) |
|---|---|---|---|---|---|---|---|
| 32x224x224 | Sports1M | link | 78.8 | 93.5 | link | 108.8 | 32.8 |
| 32x224x224 | IG-65M | link | 82.5 | 95.3 | link | 108.8 | 32.8 |
- D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri. A Closer Look at Spatiotemporal Convolutions for Action Recognition. CVPR 2018.
- D. Tran, H. Wang, L. Torresani and M. Feiszli. Video Classification with Channel-Separated Convolutional Networks. ICCV 2019.
- D. Ghadiyaram, M. Feiszli, D. Tran, X. Yan, H. Wang and D. Mahajan, Large-scale weakly-supervised pre-training for video action recognition. CVPR 2019.
VMZ is Apache 2.0 licensed, as found in the LICENSE file.
This codebase is actively supported by some members of CV team (Facebook AI): @CHJoanna, @weiyaowang, @bjuncek, @hengcv, @deeptigp, and @dutran.