multi-modality

There are 9 repositories under multi-modality topic.

haotian-liu / LLaVA
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
gpt-4 chatbot chatgpt llama multimodal llava foundation-models instruction-tuning multi-modality visual-language-learning llama-2 llama2 vision-language-model
Language:Python 19362
clip-as-service
jina-ai / clip-as-service
🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
bert bert-as-service clip-as-service clip-model cross-modal-retrieval cross-modality deep-learning image2vec multi-modality neural-search onnx openai pytorch sentence-encoding sentence2vec
Language:Python 12386
BradyFU / Awesome-Multimodal-Large-Language-Models
:sparkles::sparkles:Latest Advances on Multimodal Large Language Models
chain-of-thought in-context-learning instruction-following instruction-tuning large-language-models large-vision-language-model large-vision-language-models multi-modality multimodal-chain-of-thought multimodal-in-context-learning multimodal-instruction-tuning multimodal-large-language-models visual-instruction-tuning
11793
lucidrains / deep-daze
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun
artificial-intelligence deep-learning implicit-neural-representation multi-modality siren text-to-image transformers
Language:Python 4373
Otter
Luodian / Otter
🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
artificial-inteligence chatgpt deep-learning embodied-ai foundation-models gpt-4 instruction-tuning large-scale-models machine-learning multi-modality visual-language-learning
Language:Python 3552
InternLM / InternLM-XComposer
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
chatgpt visual-language-learning multi-modality foundation gpt-4 instruction-tuning mllm multimodal vision-language-model language-model large-language-model large-vision-language-model llm vision-transformer gpt supervised-finetuning
Language:Python 2460
swarms
kyegomez / swarms
The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework Join our Community: https://discord.com/servers/agora-999382051935506503
artificial-intelligence attention-mechanism gpt4 langchain machine-learning multi-modal-imaging multi-modality multimodal swarms transformer-models agents prompt-engineering prompt-toolkit prompting tree-of-thoughts ai chatgpt gpt4all huggingface langchain-python
Language:Python 1136
DLR-RM / 3DObjectTracking
Algorithms and Publications on 3D Object Tracking
accv2020 articulated computer-vision cvpr2022 ijcv iros2023 multi-body multi-modality object-tracking paper pose-estimation real-time rgbd tpami tracking
Language:C++ 668
OpenGVLab / Multi-Modality-Arena
Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!
chat chatbot chatgpt gradio large-language-models llms multi-modality vision-language-model vqa
Language:Python 445
kyegomez / Gemini
The open source implementation of Gemini, the model that will "eclipse ChatGPT" by Google
ai artificial-intelligence gemini gpt4 machine-learning ml multi-modality multimodla
Language:Python 411
ziqihuangg / Collaborative-Diffusion
[CVPR 2023] Collaborative Diffusion
diffusion-models face-editing face-generation image-editing image-generation latent-diffusion-models multi-modality aigc gen-ai stable-diffusion
Language:Python 395
researchmm / MM-Diffusion
[CVPR'23] MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
audio-generation content-creation diffusion-models multi-modality video-generation
Language:Python 384
kyegomez / Sophia
Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.
artificial-intelligence chatgpt deep-learning multi-modality neural-network optimizer
Language:Python 374
ZwwWayne / mmMOT
[ICCV2019] Robust Multi-Modality Multi-Object Tracking
mot iccv2019 multi-modality
Language:Python 252
DerrickWang005 / CRIS.pytorch
An official PyTorch implementation of the CRIS paper
contrastive-learning multi-modality referring-image-segmentation
Language:Python 244
xiaoachen98 / Open-LLaVA-NeXT
An open-source implementation for training LLaVA-NeXT.
chatbot chatgpt gpt-4 gpt4o large-multimodal-models llama llama3 llava llava-next multi-modality multimodal vision-language-model visual-language-learning
Language:Python 243
dvlab-research / UVTR
Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)
3d-detection multi-modality pytorch
Language:Python 227
RLHF-V / RLHF-V
[CVPR'24] RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback
chatbot gpt-4 llama multi-modality multimodal rlhf-v visual-language-learning
Language:Python 217
CVPR21Chal-SLR
jackyjsy / CVPR21Chal-SLR
This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.
sign-language-recognition sign-language-recognition-system multi-modality skeleton-features cvpr2021
Language:Python 205
yangcaoai / CoDA_NeurIPS2023
Official code for NeurIPS2023 paper: CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection
3d-detection 3d-vision multi-modality open-vocabulary artificial-intelligence deep-learning transformer detection
Language:Jupyter Notebook 175
sshh12 / multi_token
Embed arbitrary modalities (images, audio, documents, etc) into large language models.
large-context large-language-models large-multimodal-models llava llm multi-modality multimodal vision-language-model
Language:Python 170
jina-ai / rungpt
An open-source cloud-native of large multi-modal models (LMMs) serving framework.
flamingo gpt-4 large-language-models llama multi-modality opengpt transformers large-multimadality-models self-hosting llm-hosting llm-serve lmm-serve
Language:Python 150
kyegomez / the-compiler
Seed, Code, Harvest: Grow Your Own App with Tree of Thoughts!
agora artficial-intelligence autogpt chatgpt prompt-engineering tree-of-thoughts chain-of-thought deep-learning deep-learning-algorithms multi-modal-fusion multi-modality multimodal-deep-learning reinforcement-learning
Language:Python 144
Lee-Gihun / MEDIAR
(NeurIPS 2022 CellSeg Challenge - 1st Winner) Open source code for "MEDIAR: Harmony of Data-Centric and Model-Centric for Multi-Modality Microscopy"
biomedical cell-biology cell-segmentation instance-segmentation monai multi-modality neurips-2022 pytorch pytorch-implementation pytorch-segmentation vision-transformer miscroscopy multi-resolution
Language:Python 137
kyegomez / Andromeda
An all-new Language Model That Processes Ultra-Long Sequences of 100,000+ Ultra-Fast
artificial-intelligence deep-learning gpt-4 language-model large-language-models neural-networks transformer agi artificial-general-intelligence artificial-intelligence-algorithms multi-modality multimodal
Language:Python 131
dvlab-research / Prompt-Highlighter
[CVPR 2024] Prompt Highlighter: Interactive Control for Multi-Modal LLMs
llm-inference multi-modality text-generation
Language:Python 121
kyegomez / MambaByte
Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta
ai artificial-intelligence gpt4v machine-learning mamba megabyte ml multi-modality tokenizer
Language:Python 104
SsGood / MMGL
Multi-modal Graph learning for Disease Prediction (IEEE Trans. on Medical imaging, TMI2022)
pytorch multi-modality graph-learning disease-prediction transformer
Language:Jupyter Notebook 88
kyegomez / MoE-Mamba
Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Zeta
ai ml moe multi-modal-fusion multi-modality swarms
Language:Python 74
CorrI2P
rsy6318 / CorrI2P
[TCSVT] CorrI2P: Deep Image-to-Point Cloud Registration via Dense CorrespondenceThe code of CorrI2P
multimedia point-cloud registration multi-modality
Language:Python 70
kyegomez / Kosmos2.5
My implementation of Kosmos2.5 from the paper: "KOSMOS-2.5: A Multimodal Literate Model"
attention attention-is-all-you-need gpt3 gpt4 kosmos multi-modality multimodal multimodal-deep-learning opensource
Language:Python 68
rentainhe / TRAR-VQA
[ICCV 2021] Official implementation of the paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering"
vqav2 iccv2021 transformer clevr multi-modal vision-and-language visual-question-answering pytorch multi-scale-features dynamic-network attention local-and-global multi-modality visualization multi-modal-learning official
Language:Python 64
amazon-science / crossmodal-contrastive-learning
CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations, ICCV 2021
multi-modality video video-text-retrieval video-captioning computer-vision natural-language-processing transformers contrastive-learning
Language:Python 56
ecom-research / ComposeAE
Official code for WACV 2021 paper - Compositional Learning of Image-Text Query for Image Retrieval
multi-modality compositional-learning information-retrieval
Language:Python 55
OpenGVLab / LORIS
Long-Term Rhythmic Video Soundtracker, ICML2023
aigc diffusion-models multi-modality music-generation pytorch-implementation
Language:Python 54
mit-acl / deep_panther
deep-learning drones imitation-learning motion-planning multi-modality obstacle-avoidance planner trajectory-optimization
Language:C++ 51