awesome-vision-language-pretraining 
An up-to-date list of vision-language (VL) pre-training paper! Maintained by Haofan Wang (haofanwang.ai@gmail.com).
SSL+Pre-training
Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text, Google.
ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation, Baidu.
*FLAVA: A Foundational Language And Vision Alignment Model, Meta AI.
A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervision, Meta AI.
*SLIP: Self-supervision meets Language-Image Pre-training, [code], Meta AI.
*MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modeling, Amazon.
*Data Efficient Masked Language Modeling for Vision and Language, [code], Ben Gurion University.
VL-BERT: Pre-training of Generic Visual-Linguistic Representations, [code], MSRA.
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training, [code], MSRA.
UNITER: UNiversal Image-TExt Representation Learning, [code], Microsoft.
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, [code], Meta AI.
Masked Image Modeling
Masked Feature Prediction for Self-Supervised Visual Pre-Training, Meta AI.
SimMIM: A Simple Framework for Masked Image Modeling, [code], MSRA.
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers, [code], MSRA.
Masked Autoencoders Are Scalable Vision Learners, [Unofficial code], Meta AI.
iBOT: Image BERT Pre-Training with Online Tokenizer, [code], ByteDance .
BEiT: BERT Pre-Training of Image Transformers, [code], MSRA.
Masked Language Modeling
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, [code], Google.
Others
Align and Prompt: Video-and-Language Pre-training with Entity Prompts, Saleforce.
FILIP: Fine-grained Interactive Language-Image Pre-Training, [Unofficial code], Huawei.
LiT: Zero-Shot Transfer with Locked-image Text Tuning, [Unofficial code], Google.
Multimodal Few-Shot Learning with Frozen Language Models, DeepMind.
Prompting Visual-Language Models for Efficient Video Understanding, [code], SJTU.
TiP-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling, [code], CUHK.