GenjiB / LAVISH

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Vision Transformers are Parameter-Efficient Audio-Visual Learners

📗Paper|| 🏠Project Page

This is the PyTorch implementation of our paper:

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius

In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Our Method

📝 Preparation

• See each foloder for more detailed settings
• Audio-Visual Event Localization: ./AVE
• Audio-Visual Segmentation: ./AVS
• Audio-Visual Question Answering: ./AVQA

🎓 Cite

If you use this code in your research, please cite:

@InProceedings{LAVISH_CVPR2023,
author = {Lin, Yan-Bo and Sung, Yi-Lin and Lei, Jie and Bansal, Mohit and Bertasius, Gedas},
title = {Vision Transformers are Parameter-Efficient Audio-Visual Learners},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year = {2023}
}

👍 Acknowledgments

Our code is based on AVSBench and MUSIC-AVQA

✏ Future works: model checkpoints

Tasks	Checkpoints
AVE	model
AVS	model
AVQA	model