vision-language

There are 5 repositories under vision-language topic.

IDEA-Research / GroundingDINO
Official implementation of the paper "Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection"
object-detection open-world open-world-detection vision-language vision-language-transformer
Language:Python 4990
salesforce / BLIP
PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
vision-language vision-and-language-pre-training image-text-retrieval image-captioning visual-question-answering visual-reasoning vision-language-transformer
Language:Jupyter Notebook 4256
marqo
marqo-ai / marqo
Unified embedding generation and search engine. Also available on cloud - cloud.marqo.ai
deep-learning information-retrieval machinelearning vector-search tensor-search clip multi-modal search-engine transformers vision-language machine-learning semantic-search visual-search natural-language-processing hnsw knn hacktoberfest chatgpt gpt large-language-models
Language:Python 4116
OFA-Sys / Chinese-CLIP
Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
chinese clip computer-vision contrastive-loss coreml-models deep-learning image-text-retrieval multi-modal multi-modal-learning nlp pretrained-models pytorch transformers vision-and-language-pre-training vision-language
Language:Python 3612
OFA-Sys / OFA
Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
chinese image-captioning multimodal pretrained-models pretraining prompt prompt-tuning referring-expression-comprehension text-to-image-synthesis vision-language visual-question-answering
Language:Python 2323
AlibabaResearch / AdvancedLiterateMachinery
A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.
artificial-intelligence computer-vision document document-analysis document-intelligence document-recognition document-understanding documentai end-to-end-ocr multimodal multimodal-deep-learning ocr scene-text-detection scene-text-detection-recognition scene-text-recognition text-detection text-recognition vision-language vision-language-model vision-language-transformer
Language:C++ 927
mbzuai-oryx / Video-ChatGPT
"Video-ChatGPT" is a video conversation model capable of generating meaningful conversation about videos. It combines the capabilities of LLMs with a pretrained visual encoder adapted for spatiotemporal video representation. We also introduce a rigorous 'Quantitative Evaluation Benchmarking' for video-based conversational models.
chatbot clip gpt-4 llama llava mulit-modal vicuna video-chatboat video-conversation vision-language vision-language-pretraining
Language:Python 896
OFA-Sys / ONE-PEACE
A general representation model across vision, audio, language modalities. Paper: ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities
audio-language contrastive-loss foundation-models multimodal representation-learning vision-and-language vision-language vision-transformer
Language:Python 838
google-research / pix2seq
Pix2Seq codebase: multi-tasks with generative modeling (autoregressive and diffusion)
computer-vision deep-learning object-detection pix2seq tensorflow2 vision-language
Language:Jupyter Notebook 806
llm-jp / awesome-japanese-llm
日本語LLMまとめ - Overview of Japanese LLMs
awesome awesome-list awesome-lists embeddings foundation-models generative-ai generative-model generative-models japanese japanese-language language-model language-models large-language-model large-language-models llm llms multimodal vision-and-language vision-language vision-language-model
745
OpenDriveLab / DriveLM
DriveLM: Driving with Graph Visual Question Answering
autonomous-driving chain-of-thought graph-of-thoughts large-language-models llm prompt-engineering prompting tree-of-thoughts vision-language
Language:HTML 624
Algolzw / daclip-uir
[ICLR 2024] Controlling Vision-Language Models for Universal Image Restoration. 5th place in the NTIRE 2024 Restore Any Image Model in the Wild Challenge.
diffusion-models image-restoration prompt vision-language face-inpainting image-deblurring image-dehazing image-denoising image-deraining image-desnowing jpeg-artifacts-removal low-level-vision low-light-image-enhancement shadow-removal pytorch deep-learning
Language:Python 531
SunzeY / AlphaCLIP
[CVPR 2024] Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
deep-learning machine-learning vision-and-language vision-language vision-language-model vision-transformer
Language:Jupyter Notebook 489
AILab-CVC / SEED
Official implementation of SEED-LLaMA (ICLR 2024).
foundation-model multimodal vision-language
Language:Python 464
cliport / cliport
CLIPort: What and Where Pathways for Robotic Manipulation
clip robotics vision deep-learning natural-language-processing grounding vision-language manipulation pytorch rearrangement computer-vision
Language:Jupyter Notebook 417
airaria / Visual-Chinese-LLaMA-Alpaca
多模态中文LLaMA&Alpaca大语言模型（VisualCLA）
alpaca chinese llama llm lora multimodal nlp vision-language
Language:Python 355
henghuiding / Vision-Language-Transformer
[ICCV2021 & TPAMI2023] Vision-Language Transformer and Query Generation for Referring Segmentation
vision-language transformer referring-segmentation tensorflow keras iccv2021 vision-language-transformer tpami
Language:Python 334
mees / calvin
CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
natural-language-processing robotics deep-learning grounding vision-language manipulation computer-vision pytorch vision vision-and-language
Language:Python 265
mczhuge / Kaleido-BERT
💐Kaleido-BERT: Vision-Language Pre-training on Fashion Domain. (CVPR2021)
bert e-commerce fashion multimodal pre-training vision-language
Language:Python 263
HUANGLIZI / LViT
[IEEE Transactions on Medical Imaging/TMI] This repo is the official implementation of "LViT: Language meets Vision Transformer in Medical Image Segmentation"
medical-image-analysis pytorch segmentation vision-language multimodal-learning
Language:Python 249
movienet / movienet-tools
Tools for movie and video research
movie computer-vision video-understanding action-recognition deep-learning vision-language cross-modality shot-detection person-analysis
Language:C++ 249
longzw1997 / Open-GroundingDino
This is the third party implementation of the paper Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.
object-detection open-world open-world-detection vision-language
Language:Python 232
DLCV-BUAA / TinyLLaVABench
A Framework of Small-scale Large Multimodal Models
large-multimodal-models llama llava nlp tinyllama transformers vision-language
Language:Python 211
mertyg / vision-language-models-are-bows
Experiments and data for the paper "When and why vision-language models behave like bags-of-words, and what to do about it?" Oral @ ICLR 2023
blip clip compositionality multimodal pytorch vision-language
Language:Python 202
ChenDelong1999 / RemoteCLIP
🛰️ Official repository of paper "RemoteCLIP: A Vision Language Foundation Model for Remote Sensing" (IEEE TGRS)
remote-sensing vision-language contrastive-language-image-pretraining
Language:Jupyter Notebook 184
TXH-mercury / VAST
Code and Model for VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
audio-language dataset vision-language cross-modality-pretraining multimodal-foundation-model vision-audio-subtitle-text
Language:Jupyter Notebook 178
MarSaKi / VLN-BEVBert
[ICCV 2023} Official repo of "BEVBert: Multimodal Map Pre-training for Language-guided Navigation"
embodied-ai transformer vision-language
Language:Python 162
mu-cai / ViP-LLaVA
[CVPR2024] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
chatbot clip foundation-models gpt-4 gpt-4-vision llama llama2 llava multi-modal vision-language visual-prompting cvpr2024
Language:Python 148
woodfrog / vse_infty
Code for "Learning the Best Pooling Strategy for Visual Semantic Embedding", CVPR 2021
vse image-text-matching cross-modal-retrieval vision-language visual-semantic pytorch
Language:Python 147
qiantianwen / NuScenes-QA
[AAAI 2024] NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario.
autonomous-driving vision-language visual-question-answering
116
howard-hou / BagFormer
PyTorch code for BagFormer: Better Cross-Modal Retrieval via bag-wise interaction
cross-modal-retrieval image-text-retrieval vision-language
Language:Python 115
MikeWangWZHL / VidIL
Pytorch code for Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
blip clip gpt-3 msrvtt msvd vatex video-language vision-language youcook2 vlep
Language:Python 110
amazon-science / mix-generation
MixGen: A New Multi-Modal Data Augmentation
data-augmentation data-efficiency multimodal pretraining vision-language
Language:Python 104
astra-vision / PODA
[ICCV 2023] Official implementation of "PØDA: Prompt-driven Zero-shot Domain Adaptation"
deep-learning domain-adaptation feature-augmentation pytorch vision-language zero-shot semantic-segmentation computer-vision clip multi-modal scene-understanding
Language:Python 97
doc-doc / NExT-QA
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (CVPR'21)
vision-language multi-object-interaction causal-temporal-action-reasoning video-question-answering video-understanding videoqa
Language:Python 97
Clip2Protect
fahadshamshad / Clip2Protect
[CVPR 2023] Official repository of paper titled "CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via Adversarial Latent Search".
text-guidance text-guided-image-manipulation face-manipulation face-recognition privacy-protection dodging impersonation makeup-transfer stylegan vision-language
Language:Python 94