Fine-tuned CLIP models are efficient video learners
Hanoona Rasheed*, Muhammad Uzair Khattak*, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan
*Equally contributing first authors
Official implementation of the paper "Fine-tuned CLIP models are efficient video learners".
- (Nov 24, 2023)
- Interactive notebook released. Inference with ViFi-CLIP on custom videos without significant installation dependencies!
- (Feb 28, 2023)
- Paper accepted at CVPR 2023 🎉
- (Dec 6, 2022)
- Training and evaluation codes for ViFi-CLIP, along with pretrained models are released.
This work explores the capability of a simple baseline called ViFi-CLIP (Video Fine-tuned CLIP) for adapting image pretrained CLIP to video domain. The figure compares the zero-shot performance of vanilla CLIP and several of its variants adapted for videos (trained on Kinetics-400, evaluated on UCF-101 and HMDB-51). The t-SNE visualizations of video-embeddings obtained from ViFi-CLIP (4th col.) are compared with embeddings from vanilla CLIP (1st col.), individually tuned CLIP text (2nd col.) and image encoder (3rd col.) on videos, and recent state-of-the-art work, XCLIP (last col.) (∆ represents difference over XCLIP). The embeddings of ViFi-CLIP are better separable, indicating that a simple fine-tuning of CLIP is sufficient to learn suitable video-specific inductive biases, and can perform competitive to more complex approaches having dedicated components designed to model temporal information in videos.
Abstract: Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos , they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks.
- ViFi-CLIP: We formulate and show the significance of an often neglected but simple baseline for transferring image-based CLIP model to video domain. ViFi-CLIP (Video Fine-tuned CLIP) shows that a simple fine-tuning of CLIP is sufficient to learn suitable video-specific inductive biases, and can perform competitive to more complex approaches having dedicated components designed to model temporal information in videos.
- Base-to-novel generalization benchmark: We introduce base-to-novel generalization benchmark for video-domain for evaluating the generalization ability of models for video action recognition.
- Bridge and Prompt approach: We show the effectiveness of our proposed ‘bridge and prompt’ approach to first bridge the modality gap through fine-tuning followed by prompt learning in both visual and language branches of the CLIP model for low-data regimes.
NOTE: All models in our experiments below uses publicly available ViT/B-16 based CLIP model. The trained model weights against each experiment is provided in tables below.
All models are trained on Kinetics-400 and then evaluated directly on downstream datasets.
| Name (configs) | Input | HMDB-51 | UCF-101 | Kinetics-600 | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 32x224 | 49.0 | 72.9 | 62.2 | link |
| CLIP text-FT | 32x224 | 48.5 | 69.8 | 68.5 | link |
| ViFi-CLIP | 32x224 | 51.3 | 76.8 | 71.2 | link |
Here, we divide each dataset into base and novel classes. All models are trained on base classes and evaluated on both base and novel classes. Results are averaged over 3 seeds for each experiment.
| Name (configs) | Input | Base Acc. | Novel Acc. | HM | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 32x224 | 72.9 | 58.0 | 64.6 | seed1/seed2/seed3 |
| CLIP text-FT | 32x224 | 73.4 | 59.7 | 65.8 | seed1/seed2/seed3 |
| ViFi-CLIP | 32x224 | 76.4 | 61.1 | 67.9 | seed1/seed2/seed3 |
| Name (configs) | Input | Base Acc. | Novel Acc. | HM | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 32x224 | 62.6 | 47.5 | 54.0 | seed1/seed2/seed3 |
| CLIP text-FT | 32x224 | 70.0 | 51.2 | 59.1 | seed1/seed2/seed3 |
| ViFi-CLIP | 32x224 | 73.8 | 53.3 | 61.9 | seed1/seed2/seed3 |
| Name (configs) | Input | Base Acc. | Novel Acc. | HM | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 32x224 | 86.4 | 65.3 | 74.4 | seed1/seed2/seed3 |
| CLIP text-FT | 32x224 | 90.9 | 67.4 | 77.4 | seed1/seed2/seed3 |
| ViFi-CLIP | 32x224 | 92.9 | 67.7 | 78.3 | seed1/seed2/seed3 |
| Name (configs) | Input | Base Acc. | Novel Acc. | HM | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 32x224 | 9.2 | 8.5 | 8.8 | seed1/seed2/seed3 |
| CLIP text-FT | 32x224 | 12.4 | 9.5 | 10.8 | seed1/seed2/seed3 |
| ViFi-CLIP | 32x224 | 16.2 | 12.1 | 13.9 | seed1/seed2/seed3 |
ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets.
| Dataset (configs) | Input | Base Acc. | Novel Acc. | HM | Model |
|---|---|---|---|---|---|
| HMDB-51 | 32x224 | 77.1 | 54.9 | 64.1 | seed1/seed2/seed3 |
| UCF-101 | 32x224 | 95.9 | 74.1 | 83.6 | seed1/seed2/seed3 |
| SSv2 | 32x224 | 15.8 | 11.5 | 13.3 | seed1/seed2/seed3 |
Below table shows few-shot results of ViFi-CLIP for K=2, 4, 8 and 16.
| Name (configs) | Dataset | K (shots) | Input | Top-1 Acc. | Model |
|---|---|---|---|---|---|
| ViFi-CLIP | HMDB-51 | 2 | 32x224 | 57.2 | link |
| ViFi-CLIP | HMDB-51 | 4 | 32x224 | 62.7 | link |
| ViFi-CLIP | HMDB-51 | 8 | 32x224 | 64.5 | link |
| ViFi-CLIP | HMDB-51 | 16 | 32x224 | 66.8 | link |
| ViFi-CLIP | UCF-101 | 2 | 32x224 | 80.7 | link |
| ViFi-CLIP | UCF-101 | 4 | 32x224 | 85.1 | link |
| ViFi-CLIP | UCF-101 | 8 | 32x224 | 90.0 | link |
| ViFi-CLIP | UCF-101 | 16 | 32x224 | 92.7 | link |
| ViFi-CLIP | SSv2 | 2 | 32x224 | 6.2 | link |
| ViFi-CLIP | SSv2 | 4 | 32x224 | 7.4 | link |
| ViFi-CLIP | SSv2 | 8 | 32x224 | 8.5 | link |
| ViFi-CLIP | SSv2 | 16 | 32x224 | 12.4 | link |
NOTE: Few-shot results for other CLIP Fine-tuned variants are presented in our main paper (Table 3). Model weights for other variants are provided here.
ViFi-CLIP is first trained on K400 and then vision and language prompts are further fine-tuned on the downstream datasets in few-shot manner.
| Dataset (configs) | Input | K=2 | K=4 | K=8 | K=16 | Model |
|---|---|---|---|---|---|---|
| HMDB-51 | 32x224 | 63.0 | 65.1 | 69.6 | 72.0 | K=2/K=4/K=8/K=16 |
| UCF-101 | 32x224 | 91.0 | 93.7 | 95.0 | 96.4 | K=2/K=4/K=8/K=16 |
| SSv2 | 32x224 | 6.7 | 7.9 | 10.2 | 13.5 | K=2/K=4/K=8/K=16 |
| Name (configs) | FLOPS(G) | Input | Top-1 Acc. | Top-5 Acc. | Model |
|---|---|---|---|---|---|
| CLIP image-FT | 281 | 16x224 | 82.8 | 96.2 | link |
| CLIP text-FT | 281 | 16x224 | 73.1 | 91.2 | link |
| ViFi-CLIP | 281 | 16x224 | 83.9 | 96.3 | link |
For installation and other package requirements, please follow the instructions detailed in INSTALL.md.
Please follow the instructions at DATASETS.md to prepare all datasets.
For all experiments shown in above tables, we provide config files in configs folder. For example, to train ViFi-CLIP (tunes both image and text encoder) on Kinetics-400, run the following command:
python -m torch.distributed.launch --nproc_per_node=8 \
main.py -cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT
Note:
- We recommend keeping the total batch size as mentioned in respective config files. Please use
--accumulation-stepsto maintain the total batch size. Specifically, here the effective total batch size is 8(GPUs_NUM) x 4(TRAIN.BATCH_SIZE) x 16(TRAIN.ACCUMULATION_STEPS) = 512. - After setting up the datasets as instructed DATASETS.md, only argument in the config file that should be specified is data path. All other settings in config files are pre-set.
For detailed training instructions for all experimental setup, please refer to TRAIN.md.
To evaluate a model, please use a suitable config and corresponding model weights. For example, to evaluate ViFi-CLIP with 16 frames on Kinetics-400, run the command below:
python -m torch.distributed.launch --nproc_per_node=8 main.py \
-cfg configs/fully_supervised/k400/16_16_vifi_clip.yaml --output /PATH/TO/OUTPUT \
--only_test --resume /PATH/TO/CKPT --opts TEST.NUM_CLIP 4 TEST.NUM_CROP 3
If you have any questions, please create an issue on this repository or contact at uzair.khattak@mbzuai.ac.ae or hanoona.bangalath@mbzuai.ac.ae .
If you use our approach (code, model or dataset splits) in your research, please consider citing:
@inproceedings{hanoonavificlip,
title={Finetuned CLIP models are efficient video learners},
author={Rasheed, Hanoona and khattak, Muhammad Uzair and Maaz, Muhammad and Khan, Salman and Khan, Fahad Shahbaz},
booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2023}
}
Our code is based on XCLIP's repository. We sincerely thank the authors for releasing their code. If you use our model and code, please consider citing XCLIP as well.