vision-language-pretraining

There are 10 repositories under vision-language-pretraining topic.

deepseek-ai / Janus
Janus-Series: Unified Multimodal Understanding and Generation Models
any-to-any foundation-models llm multimodal unified-model vision-language-pretraining
Language:Python 17543
salesforce / LAVIS
LAVIS - A One-stop Library for Language-Vision Intelligence
deep-learning deep-learning-library image-captioning salesforce vision-and-language vision-framework vision-language-pretraining vision-language-transformer visual-question-anwsering multimodal-datasets multimodal-deep-learning
Language:Jupyter Notebook 10889
deepseek-ai / DeepSeek-VL
DeepSeek-VL: Towards Real-World Vision-Language Understanding
vision-language-model vision-language-pretraining foundation-models
Language:Python 3958
DAMO-NLP-SG / Video-LLaMA
[EMNLP 2023 Demo] Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
blip2 cross-modal-pretraining large-language-models llama minigpt4 multi-modal-chatgpt video-language-pretraining vision-language-pretraining
Language:Python 3064
mbzuai-oryx / Video-ChatGPT
[ACL 2024 🔥] Video-ChatGPT is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
chatbot clip gpt-4 llama llava mulit-modal vicuna vision-language vision-language-pretraining video-chatboat video-conversation
Language:Python 1434
Sense-GVT / DeCLIP
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
big-model clip image-text multi-model self-supervised vision-language-pretraining zero-shot
Language:Python 650
mbzuai-oryx / VideoGPT-plus
Official Repository of paper VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding
chatbot clip dual-encoder gpt4 gpt4o image-encoder llama3 llava multimodal phi-3-mini vicuna video-chatbot video-conversation video-encoder vision-language vision-language-pretraining
Language:Python 286
TXH-mercury / VALOR
[TPAMI2024] Codes and Models for VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
audio-language-pretraining audiovisual-language-pretraining multimodal-representation-learning vision-language-pretraining
Language:Python 284
sail-sg / ptp
[CVPR2023] The code for 《Position-guided Text Prompt for Vision-Language Pre-training》
cross-modality vision-language-pretraining vlp
Language:Python 152
Surrey-UP-Lab / RegionSpot
Recognize Any Regions
vision-foundation-model auto-labeling instance-segmentation multimodal-representation-learning object-detection open-vocabulary open-world vision-language-model vision-language-pretraining zero-shot vision-language-foundation-model
Language:Python 121
jusiro / FLAIR
[MedIA'25] FLAIR: A Foundation LAnguage-Image model of the Retina for fundus image understanding.
foundation-models fundus-image-analysis medical-imaging vision-language-pretraining
Language:Python 115
vgthengane / Continual-CLIP
Official repository for "CLIP model is an Efficient Continual Learner".
clip continual-learning vision-language-pretraining foundational-models baseline
Language:Python 93
ArrowLuo / SegCLIP
PyTorch implementation of ICML 2023 paper "SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation"
open-vocabulary semantic-segmentation zero-shot-semantic-segmentation open-vocabulary-semantic-segmentation transfer-learning vision-language-pretraining contrastive-learning
Language:Python 90
HieuPhan33 / CVPR2024_MAVL
Multi-Aspect Vision Language Pretraining - CVPR2024
medical-vision-and-language-pretraining vision-language-model vision-language-pretraining zero-shot-classification zero-shot-segmentation
Language:Python 82
marslanm / Multimodality-Representation-Learning
This repository provides a comprehensive collection of research papers focused on multimodal representation learning, all of which have been cited and discussed in the survey just accepted https://dl.acm.org/doi/abs/10.1145/3617833 .
cross-modal multimodal-datasets multimodal-deep-learning multimodal-pre-trained-model transformer-models vision-language-pretraining multimodal-applications multimodal-pretext
72
Zoky-2020 / SGA
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models. [ICCV 2023 Oral]
adversarial-attack vision-language-pretraining
Language:Python 59
megvii-research / protoclip
📍 Official pytorch implementation of paper "ProtoCLIP: Prototypical Contrastive Language Image Pretraining" (IEEE TNNLS)
contrastive-learning self-supervised-learning vision-language-pretraining
Language:Python 52
TXH-mercury / COSA
[ICLR2024] Codes and Models for COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
video-captioning video-language-pretrainng video-qa video-retrieval vision-language-pretraining
Language:Python 43
TencentARC / FLM
Accelerating Vision-Language Pretraining with Free Language Modeling (CVPR 2023)
language-modeling vision-language-pretraining
Language:Python 32
alinlab / b2t
Bias-to-Text: Debiasing Unknown Visual Biases through Language Interpretation
explainable-ai vision-language-pretraining bias-and-fairness
Language:Python 31
jaisidhsingh / LoRA-CLIP
Easy wrapper for inserting LoRA layers in CLIP.
image-text-matching lora low-rank-adaptation multimodal multimodal-deep-learning parameter-efficient-tuning vision-language-pretraining
Language:Python 31
yiren-jian / BLIText
[NeurIPS 2023] Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
multimodal-deep-learning vision-language-pretraining vision-language-transformer
Language:Python 25
YyzHarry / vlm-fairness
[Science Advances] Demographic Bias of Vision-Language Foundation Models in Medical Imaging
algorithmic-fairness bias-mitigation fairness foundation-models medical-imaging ood-generalization subpopulation vision-language-model vision-language-pretraining
Language:Python 21
omipan / svl_adapter
SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
self-supervised-learning vision-language-pretraining
Language:Python 20
adarobustness / adaptation_robustness
Evaluate robustness of adaptation methods on large vision-language models
adaptation parameter-efficient-tuning robustness vision-language-pretraining
Language:Shell 19
ChenDelong1999 / ITRA
A codebase for flexible and efficient Image Text Representation Alignment
computer-vision deep-learning multimodal-learning pytorch vision-language-pretraining
Language:Python 18
unitaryai / VTC
VTC: Improving Video-Text Retrieval with User Comments
multimodal-deep-learning video-text-retrieval video-understanding vision-language-pretraining vision-language-transformer comments
Language:Python 13
LooperXX / ManagerTower
Code for ACL 2023 Oral Paper: ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning
multi-modal-learning vision-language vision-language-pretraining vision-language-learning
Language:Python 11
ahmdtaha / distributed_sigmoid_loss
Unofficial implementation for Sigmoid Loss for Language Image Pre-Training
contrastive-learning distributed-data-parallel multimodal-deep-learning python3 pytorch self-supervised-learning unsupervised-learning vision-and-language vision-language vision-language-pretraining vision-transformer
Language:Python 10
xmed-lab / FD-SOS
MICCAI 2024 Oral: Vision-Language Open-Set Detectors for Bone Fenestration and Dehiscence Detection from Intraoral Images
miccai2024 object-detection object-detector open-set-object-detection teeth vision-language vision-language-pretraining
Language:Python 7
BUAADreamer / CCRK
[KDD 2024] Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning
cross-lingual cross-lingual-retrieval cross-modal cross-modal-retrieval iglue image-text-retrieval image-text-search kdd2024 mscoco multi30k retrieval swin-transformer vision-language-pretraining wit xflickrco xlm-roberta
Language:Python 5
PGSmall / clip-pgs
Official code for CVPR2025 "Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection"
clip masked-image-modeling vision-language-pretraining vlms
Language:Python 5
unitaryai / VTC-dataset
dataset video-text-retrieval video-understanding vision-language-pretraining vision-language-dataset
Language:Python 0