hzhang57 / Awesome-CLIP

Awesome list for research on CLIP (Contrastive Language-Image Pre-Training).

Awesome CLIP

This repo collects the research resources based on CLIP (Contrastive Language-Image Pre-Training) proposed by OpenAI. If you would like to contribute, please open an issue.

CLIP

• Learning Transferable Visual Models From Natural Language Supervision [paper][code]
• CLIP: Connecting Text and Images [blog]
• Multimodal Neurons in Artificial Neural Networks [blog]

Training

• OpenCLIP (3rd-party, PyTorch) [code]
• Train-CLIP (3rd-party, PyTorch) [code]
• Paddle-CLIP (3rd-party, PaddlePaddle) [code]

Applications

GAN

• VQGAN-CLIP [code]
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [paper][code]
• CLIP Guided Diffusion [code]
• CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions [paper]
• TargetCLIP: Image-Based CLIP-Guided Essence Transfer [paper][code]
• DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation [paper][code]

Object Detection

• Roboflow Zero-shot Object Tracking [code]
• Zero-Shot Detection via Vision and Language Knowledge Distillation [paper][code]
• Crop-CLIP [code]
• Detic: Detecting Twenty-thousand Classes using Image-level Supervision [paper][code]
• CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks [paper]
• SLIP: Self-supervision meets Language-Image Pre-training [paper][code]
• ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension [paper][code]

Information Retrieval

• Unsplash Image Search [code]
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval [paper][code]
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling [paper][code]
• Natural Language YouTube Search [code]
• CLIP-as-service: Embed images and sentences into fixed-length vectors with CLIP [doc][code]
• clip-retrieval [code]

Video Understanding

• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding [code]
• FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks [paper][code]

Image Captioning

• CLIP prefix captioning [code]
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning [paper]

Image Editing

• HairCLIP: Design Your Hair by Text and Reference Image [code]
• CLIPstyler: Image Style Transfer with a Single Text Condition [paper][code]
• CLIPasso: Semantically-Aware Object Sketching [paper][code]

Text-to-3D Generation

• CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation [paper]
• Text2Mesh: Text-Driven Neural Stylization for Meshes [paper][code]
• CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP [paper]
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders [paper]
• CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields [paper][code]
• MotionCLIP: Exposing Human Motion Generation to CLIP Space [paper][code]

Representation Learning

• Wav2CLIP: Learning Robust Audio Representations From CLIP [code]
• CLIP-Lite: Information Efficient Visual Representation Learning from Textual Annotation [paper]
• RgionCLIP: Region-based Language-Image Pretraining [Paper]
• CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification [paper]
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [paper][code]

3D Recognition

• PointCLIP: Point Cloud Understanding by CLIP [Paper][code]

Language Tasks

• CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment [paper]

Object Navigation

• CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration [paper]

Audio

• AudioCLIP: Extending CLIP to Image, Text and Audio [code]

Text-to-Image Generation

• Big Sleep: A simple command line tool for text to image generation [code]
• Deep Daze: A simple command line tool for text to image generation [code]

Localization

• Adapting CLIP For Phrase Localization Without Further Training [paper][code]

Others

• Multilingual-CLIP [code]
• CLIP (With Haiku + Jax!) [code]
• CLIP-Event: Connecting Text and Images with Event Structures [paper][code]
• How Much Can CLIP Benefit Vision-and-Language Tasks? [paper]
• DeCLIP: Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm [paper][code]
• Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision [paper][code]
• CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning [paper][code]

Acknowledgment

Inspired by Awesome Visual-Transformer.