multi-modal-learning

There are 16 repositories under multi-modal-learning topic.

mlfoundations / open_clip
An open source implementation of CLIP.
computer-vision contrastive-loss deep-learning language-model multi-modal-learning pretrained-models pytorch zero-shot-classification
Language:Python 12901
OFA-Sys / Chinese-CLIP
Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
chinese clip computer-vision contrastive-loss coreml-models deep-learning image-text-retrieval multi-modal multi-modal-learning nlp pretrained-models pytorch transformers vision-and-language-pre-training vision-language
Language:Jupyter Notebook 5599
lyuchenyang / Macaw-LLM
Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration
deep-learning language-model machine-learning multi-modal-learning natural-language-processing neural-networks
Language:Python 1585
NVlabs / prismer
The implementation of "Prismer: A Vision-Language Model with Multi-Task Experts".
image-captioning language-model multi-modal-learning multi-task-learning vision-language-model vision-and-language vqa
Language:Python 1304
lucidrains / x-clip
A concise but complete implementation of CLIP with various experimental improvements from recent papers
artificial-intelligence deep-learning contrastive-learning zero-shot-learning multi-modal-learning
Language:Python 718
jokieleung / awesome-visual-question-answering
A curated list of Visual Question Answering(VQA)(Image/Video Question Answering),Visual Question Generation ,Visual Dialog ,Visual Commonsense Reasoning and related area.
attention-networks awesome-list multi-modal multi-modal-learning vqa
667
InternRobotics / EmbodiedScan
[CVPR 2024 & NeurIPS 2024] EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI
3d-vision computer-vision multi-modal-learning robotics
Language:Python 638
zeta
kyegomez / zeta
Build high-performance AI models with modular building blocks
artificial-intelligence multi-modal transformers deep-learning gpt4 llama2 multi-agent-systems multi-modal-learning multi-platform pytorch speech-recognition transformer longnet
Language:Python 562
zjukg / KG-MM-Survey
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
awsome-list cross-modal-retrieval entity-alignment entity-linking image-classification image-generation information-extraction knowledge-graph knowledge-graph-embeddings large-language-models multi-modal-fusion multi-modal-knowledge-graph multi-modal-learning paper-list survey surveys visual-question-answering awsome
458
DmitryRyumin / CVPR-2023-24-Papers
CVPR 2023-2024 Papers: Dive into advanced research presented at the leading computer vision conference. Keep up to date with the latest developments in computer vision and deep learning. Code included. ⭐ support visual intelligence development!
action-recognition autonomous-driving biometrics computer-vision cvpr cvpr2023 datasets deep-learning face-recognition gesture-recognition image-synthesis medical-image-processing multi-modal-learning pattern-recognition scene-analysis segmentation self-supervised-learning shape-analysis video-synthesis cvpr2024
Language:Python 457
zhengli97 / PromptKD
[CVPR 2024] Official PyTorch Code for "PromptKD: Unsupervised Prompt Distillation for Vision-Language Models"
cvpr2024 multi-modal-learning prompt-learning vision-language-model knowledge-distillation clip
Language:Python 336
Ysz2022 / NeRCo
[ICCV 2023] Implicit Neural Representation for Cooperative Low-light Image Enhancement
low-light-image low-light-image-enhancement neural-representation multi-modal-learning iccv iccv2023
Language:Python 254
moabarar / nemar
[CVPR2020] Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation
multimodal registartion stn image-to-image-translation multi-modal multi-modal-learning affine-transformation deformable-transformation deep-learning cnn pytorch image-registration cvpr2020 multimodal-image-registration
Language:Python 189
GuanRunwei / Achelous
The official repository of Achelous and Achelous++
multi-modal-fusion multi-modal-learning multi-task-learning object-detection object-tracking panoptic-perception point-cloud-segmentation semantic-segmentation 4d-mmwave-radar
Language:Python 159
huggingface / chug
Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.
computer-vision dataloading datasets distributed-training document-understanding multi-modal-learning pdf-document webdataset
Language:Python 159
qizekun / ReCon
[ICML 2023] Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
3d-point-clouds multi-modal-learning representation-learning self-supervised-learning
Language:Python 147
wjun0830 / CGDETR
Official pytorch repository for CG-DETR "Correlation-guided Query-Dependency Calibration in Video Representation Learning for Temporal Grounding"
computer-vision detection-transformer detr highlight-detection moment-retrieval multi-modal-learning pytorch temporal-grounding text-video-retrieval video-grounding video-summarization video-understanding
Language:Python 141
shikras / d-cube
A detection/segmentation dataset with labels characterized by intricate and flexible expressions. "Described Object Detection: Liberating Object Detection with Flexible Expressions" (NeurIPS 2023).
multi-modal-learning object-detection referring-expression-comprehension vision-language dataset open-vocabulary-detection
Language:Python 138
kkakkkka / ETRIS
[ICCV-2023] The official code of Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation
deep-learning deep-neural-networks machine-learning-algorithms multi-modal-fusion multi-modal-learning referring-image-segmentation segmentation
Language:Python 136
924973292 / EDITOR
【CVPR2024】Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification
cvpr2024 frequency-analysis msvr310 multi-modal multi-modal-learning person-reid reid rgbnt100 rgbnt201 token-selection vehicle-reidentification
Language:Python 111
ExplainableML / flair
[CVPR 2025] FLAIR: VLM with Fine-grained Language-informed Image Representations
clip contrastive-learning multi-modal-learning vision-language-model
Language:Python 110
zhengli97 / ATPrompt
[ICCV 2025] Official PyTorch Code for "Advancing Textual Prompt Learning with Anchored Attributes"
image-classification multi-modal-learning prompt-learning zero-shot-classification
Language:Python 104
likyoo / Multimodal-Remote-Sensing-Toolkit
A python tool to perform deep learning experiments on multimodal remote sensing data.
python pytorch remote-sensing multi-modal-learning
Language:Python 94
WillDreamer / Aurora
[NeurIPS2023] Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
multi-modal-learning parameter-efficient-tuning
Language:Python 88
RAIVNLab / sugar-crepe
[NeurIPS 2023] A faithful benchmark for vision-language compositionality
benchmark deep-learning multi-modal-learning pytorch vision-and-language
Language:Python 86
924973292 / Awesome-Multi-Modal-Object-Re-Identification
Welcome to the Awesome Multi-Modal Object Re-Identification Repository! This repository is dedicated to curating and sharing the latest methods, datasets, and resources focused specifically on the domain of multi-modal object re-identification. It brings together cutting-edge research, tools, and papers aimed at advancing the study and application.
awesome code-list missing-modal-retrieval multi-modal-learning multi-modal-object-re-identification paper-list person-reidentification vehicle-reidentification
84
RL4M / MRM-pytorch
An official implementation of Advancing Radiograph Representation Learning with Masked Record Modeling (ICLR'23)
chest-xray-images multi-modal-learning pre-trained-model representation-learning self-supervised-learning
Language:Python 82
vishalned / MMEarth-data
This repository contains code to download data for the preprint "MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning"
computer-vision earth-observation multi-modal-learning remote-sensing representation-learning
Language:Python 80
HyperDenseNet_pytorch
josedolz / HyperDenseNet_pytorch
Pytorch version of the HyperDenseNet deep neural network for multi-modal image segmentation
hyperdensenet deep-learning 3d-convolutional-network 3d-cnn medical-image-processing medical-image-segmentation multi-modal-imaging multi-modal-learning segmentation image-segmentation pytorch pytorch-cnn
Language:Python 79
rinnakk / japanese-clip
Japanese CLIP by rinna Co., Ltd.
clip cloob japanese pretrained-models vision language-model multi-modal-learning
Language:Python 74
3dlg-hcvc / DuoduoCLIP
[ICLR 2025] Duoduo CLIP: Efficient 3D Understanding with Multi-View Images
3d-classification 3d-shape-retrieval 3d-understanding clip multi-modal-learning pytorch
Language:Python 71
yihedeng9 / STIC
Enhancing Large Vision Language Models with Self-Training on Image Comprehension.
llava llm-finetuning multi-modal vision-language-model mistral-7b multi-modal-learning
Language:Python 70
ttgeng233 / UnAV
Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline (CVPR 2023)
audio-visual-events audio-visual-learning multi-modal-learning
Language:Python 69
rentainhe / TRAR-VQA
[ICCV 2021] Official implementation of the paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering"
vqav2 iccv2021 transformer clevr multi-modal vision-and-language visual-question-answering pytorch multi-scale-features dynamic-network attention local-and-global multi-modality visualization multi-modal-learning official
Language:Python 67
UNITES-Lab / Flex-MoE
[NeurIPS 2024 Spotlight] Code for the paper "Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts"
mixture-of-experts multi-modal-learning
Language:Python 63
zhjohnchan / awesome-vision-and-language-pretraining
A curated list of vision-and-language pre-training (VLP). :-)
multi-modal-learning pre-training vision-and-language-pre-training
59