multi-modal

There are 31 repositories under multi-modal topic.

OpenBMB / MiniCPM-V
MiniCPM-V 4.5: A GPT-4o Level MLLM for Single Image, Multi Image and High-FPS Video Understanding on Your Phone
minicpm minicpm-v multi-modal
Language:Python 22202
agentscope-ai / agentscope
AgentScope: Agent-Oriented Programming for Building LLM Applications
agent chatbot large-language-models llm llm-agent mcp multi-agent multi-modal react-agent
Language:Python 13799
OpenGVLab / InternVL
[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型
gpt gpt-4o gpt-4v image-classification image-text-retrieval llm multi-modal semantic-segmentation video-classification vision-language-model vit-22b vit-6b
Language:Python 9457
activeloopai / deeplake
Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
datasets deep-learning machine-learning data-science pytorch tensorflow python ai ml mlops computer-vision cv image-processing datalake langchain llm large-language-models vector-database vector-search multi-modal
Language:Python 8888
TEN-framework / ten-framework
Open-source framework for conversational voice AI agents
ai multi-modal real-time video voice
Language:C 8543
modelscope
modelscope / modelscope
ModelScope: bring the notion of Model-as-a-Service to life.
cv deep-learning machine-learning multi-modal nlp python science speech
Language:Python 8458
zai-org / CogVLM
a state-of-the-art-level open visual language model | 多模态预训练模型
cross-modality language-model multi-modal pretrained-models visual-language-models
Language:Python 6689
big-AGI
enricoros / big-AGI
AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. Includes AI personas, AGI functions, world-class Beam multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.
agi gpt ai-agents ai-suite ai-workspace anthropic-api deepseek-api gemini-api gpt-5 multi-modal multi-model nano-banana openai-responses-api perplexity-api sonnet-4-5 xai-api
Language:TypeScript 6682
lucidrains / DALLE-pytorch
Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch
artificial-intelligence attention-mechanism deep-learning multi-modal text-to-image transformers
Language:Python 5630
OFA-Sys / Chinese-CLIP
Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.
chinese clip computer-vision contrastive-loss coreml-models deep-learning image-text-retrieval multi-modal multi-modal-learning nlp pretrained-models pytorch transformers vision-and-language-pre-training vision-language
Language:Jupyter Notebook 5610
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data data-analysis data-pipeline data-processing data-science data-visualization foundation-models instruction-tuning large-language-models llm llms multi-modal pre-training synthetic-data
Language:Python 5502
valhalla / valhalla
Open Source Routing Engine for OpenStreetMap
astar dijkstra directions isochrones multi-modal openstreetmap routing routing-engine tiled traveling-salesman
Language:C++ 5161
marqo
marqo-ai / marqo
Unified embedding generation and search engine. Also available on cloud - cloud.marqo.ai
deep-learning information-retrieval machinelearning vector-search tensor-search clip multi-modal search-engine transformers vision-language machine-learning semantic-search visual-search natural-language-processing hnsw knn hacktoberfest chatgpt gpt large-language-models
Language:Python 4986
VectorSpaceLab / OmniGen
OmniGen: Unified Image Generation. https://arxiv.org/pdf/2409.11340
diffusion image image-edit image-generation multi-modal multi-task
Language:Jupyter Notebook 4284
zjunlp / DeepKE
[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction
knowledge-graph relation-extraction chinese named-entity-recognition attribute-extraction low-resource document-level information-extraction pytorch deepke ner nlp few-shot prompt deep-learning kg multi-modal lightner knowprompt instructie
Language:Python 4182
zai-org / VisualGLM-6B
Chinese and English multimodal conversational language model | 多模态中英双语对话语言模型
chatglm-6b gpt multi-modal
Language:Python 4170
SciSharp / LLamaSharp
A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.
chatbot gpt llama llama-cpp llama2 llama3 llamacpp llava llm multi-modal semantic-kernel
Language:C# 3427
PKU-YuanGroup / Video-LLaVA
【EMNLP 2024🔥】Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
instruction-tuning large-vision-language-model multi-modal
Language:Python 3397
open-compass / VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
chatgpt claude clip computer-vision evaluation gemini gpt gpt-4v gpt4 large-language-models llava llm multi-modal openai openai-api pytorch qwen vit vqa
Language:Python 3351
docarray
docarray / docarray
Represent, send, store and search multimodal data
docarray data-structures multimodal cross-modal neural-search deep-learning nested-data qdrant weaviate nearest-neighbor-search protobuf elasticsearch dataclass multi-modal semantic-search machine-learning pytorch fastapi pydantic
Language:Python 3111
dvlab-research / LISA
Project Page for "LISA: Reasoning Segmentation via Large Language Model"
large-language-model llm multi-modal segmentation
Language:Python 2487
zai-org / CogVLM2
GPT4V-level open-source multi-modal model based on Llama3-8B
cogvlm pretrained-models language-model multi-modal
Language:Python 2420
PKU-YuanGroup / MoE-LLaVA
【TMM 2025🔥】 Mixture-of-Experts for Large Vision-Language Models
large-vision-language-model mixture-of-experts moe multi-modal
Language:Python 2270
tangxyw / RecSysPapers
推荐/广告/搜索领域工业界经典以及最前沿论文集合。A collection of industry classics and cutting-edge papers in the field of recommendation/advertising/search.
calibration causal-inference cold-start contrastive-learning debias distillation diverse fairness match papers rank re-rank recommendation-system feedback-delay pre-rank look-alike multi-scenario multi-task reinforcement-learning multi-modal
Language:Python 1983
Kav-K / GPTDiscord
A robust, all-in-one GPT interface for Discord. ChatGPT-style conversations, image generation, AI-moderation, custom indexes/knowledgebase, youtube summarizer, and more!
artificial-intelligence asyncio gpt3 help-wanted openai openai-api python dalle2 embeddings extractive-question-answering pinecone moderator-bot hacktoberfest github digitalocean multi-modal collaborate chatbot code-interpreter discord
Language:Python 1852
OpenMotionLab / MotionGPT
[NeurIPS 2023] MotionGPT: Human Motion as a Foreign Language, a unified motion-language generation model using LLMs
3d-generation chatgpt gpt language-model motion motion-generation text-driven text-to-motion motiongpt multi-modal
Language:Python 1791
fastRAG
IntelLabs / fastRAG
Efficient Retrieval Augmentation and Generation Framework
benchmark colbert diffusion generative-ai information-retrieval knowledge-graph llm multi-modal nlp question-answering semantic-search sentence-transformers summarization transformers
Language:Python 1742
bytedance / SALMONN
SALMONN family: A suite of advanced multi-modal LLMs
audio audio-processing audio-visual-understanding bytedance iclr2024 icml-2024 large-language-models multi-modal music research speech speech-recognition tsinghua-university video video-understanding
1352
DirtyHarryLYL / Transformer-in-Vision
Recent Transformer-based CV and related works.
computer-vision deep-learning multi-modal paper self-attention transformer vision-transformers visual-language
1334
vercel / modelfusion
The TypeScript library for building AI applications.
chatbot gpt-3 javascript js llm openai ts typescript whisper ai embedding huggingface dall-e stable-diffusion llamacpp artificial-intelligence claude multi-modal ollama mistral
Language:TypeScript 1302
MedMNIST
MedMNIST / MedMNIST
[pip install medmnist] 18x Standardized Datasets for 2D and 3D Biomedical Image Classification
dataset benchmark automl mnist medical medical-image-analysis medmnist multi-modal decathlon medical-imaging medical-image-computing deep-learning machine-learning 3d 2d classification few-shot-learning pytorch federated-learning
Language:Python 1280
lucidrains / transfusion-pytorch
Pytorch implementation of Transfusion, "Predict the Next Token and Diffuse Images with One Multi-Modal Model", from MetaAI
artificial-intelligence attention deep-learning flow-matching multi-modal transformers
Language:Python 1259
Tebmer / Awesome-Knowledge-Distillation-of-LLMs
This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
alignment compression data-augmentation data-synthesis feedback instruction-following kd knowledge-distillation large-language-model llm multi-modal self-distillation self-training supervised-finetuning survey
1203
OpenBMB / VisRAG
Parsing-free RAG supported by VLMs
rag retrieval retrieval-augmented-generation vision-language-model multi-modal multi-modality document-retrieval document-understanding
Language:Python 853
PKU-YuanGroup / LanguageBind
【ICLR 2024🔥】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
language-central multi-modal pretraining zero-shot
Language:Python 845
AnswerDotAI / byaldi
Use late-interaction multi-modal models such as ColPali in just a few lines of code.
colbert colpali multi-modal nlp rag reranking retrieval
Language:Python 829