multimodal

There are 71 repositories under multimodal topic.

anything-llm
Mintplex-Labs / anything-llm
The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.
ai-agents custom-ai-agents deepseek kimi llama3 llm lmstudio local-llm localai mcp mcp-servers moonshot multimodal no-code ollama qwen3 rag vector-database web-scraping
Language:JavaScript 50798
haotian-liu / LLaVA
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
chatbot chatgpt foundation-models gpt-4 instruction-tuning llama llama-2 llama2 llava multi-modality multimodal vision-language-model visual-language-learning
Language:Python 23910
microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
beit beit-3 bitnet deepnet document-ai foundation-models kosmos kosmos-1 layoutlm layoutxlm llm minilm mllm multimodal nlp pre-trained-model textdiffuser trocr unilm xlm-e
Language:Python 21818
serve
jina-ai / serve
☁️ Build multimodal AI applications with cloud-native stack
cloud-native cncf deep-learning docker fastapi framework generative-ai grpc jaeger kubernetes llmops machine-learning microservice mlops multimodal neural-search opentelemetry orchestration pipeline prometheus
Language:Python 21780
bytedance / UI-TARS-desktop
The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra
agent agent-tars browser-use computer-use gui-agent gui-operator mcp mcp-server multimodal tars ui-tars vision vlm
Language:TypeScript 19429
deepseek-ai / Janus
Janus-Series: Unified Multimodal Understanding and Generation Models
any-to-any foundation-models llm multimodal unified-model vision-language-pretraining
Language:Python 17603
NVIDIA-NeMo / NeMo
A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
machine-translation speaker-recognition asr tts generative-ai multimodal deeplearning neural-networks speaker-diariazation speech-translation speech-synthesis large-language-models
Language:Python 16057
mediar-ai / screenpipe
AI app store powered by 24/7 desktop history. open source | 100% local | dev friendly | 24/7 screen, mic recording
agents agi ai computer-vision llm machine-learning ml multimodal vision
Language:TypeScript 15903
modelscope / ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 500+ LLMs (Qwen3, Qwen3-MoE, Llama4, GLM4.5, InternLM3, DeepSeek-R1, ...) and 200+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, Llava, GLM4v, Phi4, ...) (AAAI 2025).
llm lora llama sft multimodal peft internvl liger deepseek-r1 embedding grpo open-r1 megatron llama4 qwen3 reranker qwen3-next moe qwen3-vl qwen3-omni
Language:Python 10939
rerun
rerun-io / rerun
Visualize streams of multimodal data. Free, fast, easy to use, and simple to integrate. Built in Rust.
computer-vision cpp multimodal python robotics rust visualization
Language:Rust 9520
apache / seatunnel
SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.
data-integration high-performance offline real-time apache batch cdc change-data-capture data-ingestion elt streaming embeddings llm multimodal
Language:Java 8867
BentoML
bentoml / BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
ai-inference deep-learning generative-ai inference-platform llm llm-inference llm-serving llmops machine-learning ml-engineering mlops model-inference-service model-serving multimodal python
Language:Python 8186
SkalskiP / courses
This repository is a curated collection of links to various courses and resources about Artificial Intelligence (AI)
computer-vision deep-learning deep-neural-networks machine-learning mlops multimodal transformers tutorial natural-language-processing nlp generative-model stable-diffusion
Language:Python 6225
X-PLUG / MobileAgent
Mobile-Agent: The Powerful GUI Agent Family
agent mllm mobile-agents multimodal multimodal-large-language-models multimodal-agent android app gui mobile automation copilot
Language:Python 6199
swyxio / ai-notes
notes for software engineers getting up to speed on new AI developments. Serves as datastore for https://latent.space writing, and product brainstorming, but has cleaned up canonical references under the /Resources folder.
ai gpt gpt-3 multimodal openai prompt-engineering stable-diffusion
Language:HTML 6088
om-ai-lab / VLM-R1
Solve Visual Understanding with Reinforced VLMs
deepseek-r1 grpo llm multimodal multimodal-r1 qwen r1-zero reinforcement-learning vlm vlm-r1
Language:Python 5681
facebookresearch / mmf
A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
captioning deep-learning dialog hateful-memes multi-tasking multimodal pretrained-models pytorch textvqa vqa
Language:Python 5598
pyspur
PySpur-Dev / pyspur
A visual playground for agentic workflows: Iterate over your agents 10x faster
agent agents ai builder deepseek framework gemini graph human-in-the-loop llm llms loops multimodal ollama python rag reasoning tool trace workflow
Language:TypeScript 5577
swarms
kyegomez / swarms
The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. Website: https://swarms.ai
artificial-intelligence attention-mechanism gpt4 langchain machine-learning multi-modal-imaging multi-modality multimodal swarms transformer-models agents prompt-engineering prompt-toolkit prompting tree-of-thoughts ai chatgpt gpt4all huggingface langchain-python
Language:Python 5374
InternLM / xtuner
A Next-Generation Training Engine Built for Ultra-Large MoE Models
agent deepseek-v3 gpt-oss intern-s1 internvl kimi-k2 llm multimodal qwen3-moe reinforcement-learning
Language:Python 4967
firebase / genkit
Open-source framework for building AI-powered apps in JavaScript, Go, and Python, built and used in production by Google
agents ai embedders genkit llm multimodal rag vector-database
Language:TypeScript 4933
Daft
Eventual-Inc / Daft
Distributed query engine providing simple and reliable data processing for any modality and scale
artificial-intelligence big-data data-engineering distributed-computing machine-learning multimodal python rust
Language:Rust 4677
PKU-Alignment / align-anything
Align Anything: Training All-modality Model with Feedback
large-language-models multimodal rlhf chameleon dpo vision-language-model
Language:Jupyter Notebook 4584
kyegomez / tree-of-thoughts
Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%
artificial-intelligence chatgpt gpt4 multimodal prompt-engineering deep-learning prompt prompt-learning prompt-tuning
Language:Python 4554
luban-agi / Awesome-AIGC-Tutorials
Curated tutorials and resources for Large Language Models, AI Painting, and more.
ai aigc awesome chatgpt courses-resource deep-learning llm midjourney multimodal nlp prompt-engineering stable-diffusion tutorials
4382
rom1504 / img2dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
big-data dataset deep-learning download-images image image-dataset multimodal
Language:Python 4208
Fengshenbang-LM
IDEA-CCNL / Fengshenbang-LM
Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。
chinese-nlp pretrained-models pytorch distributed-training transformers aigc multimodal
Language:Python 4150
discoart
jina-ai / discoart
🪩 Create Disco Diffusion artworks in one line
creative-ai disco-diffusion cross-modal dalle generative-art multimodal diffusion prompts midjourney imgen discodiffusion creative-art clip-guided-diffusion latent-diffusion stable-diffusion
Language:Python 3833
open-mmlab / mmpretrain
OpenMMLab Pre-training Toolbox and Benchmark
image-classification resnet mobilenet pytorch deep-learning swin-transformer beit clip constrastive-learning convnext mae masked-image-modeling moco pretrained-models self-supervised-learning vision-transformer multimodal
Language:Python 3761
NExT-GPT / NExT-GPT
Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model
chatgpt foundation-models gpt-4 instruction-tuning large-language-models llm mllm multi-modal-chatgpt multimodal visual-language-learning
Language:Python 3579
atfortes / Awesome-LLM-Reasoning
From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 🍓
awesome chain-of-thought chatgpt cot deepseek deepseek-r1 gpt gpt-4o in-context-learning language-models mllm multimodal openai-o1 papers prompt prompt-engineering reasoning strawberry
3408
morphik-org / morphik-core
The most accurate document search and store for building AI apps
artificial-intelligence cache-augmented-generation colpali database litellm multimodal rag rules-based-ingestion
Language:Python 3352
EvolvingLMMs-Lab / lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
agi evaluation large-language-models multimodal
Language:Python 3263
OpenGVLab / InternGPT
InternGPT (iGPT) is an open source demo platform where you can easily showcase your AI models. Now it supports DragGAN, ChatGPT, ImageBind, multimodal chat like GPT-4, SAM, interactive image editing, etc. Try it at igpt.opengvlab.com (支持DragGAN、ChatGPT、ImageBind、SAM的在线Demo系统)
chatgpt foundation-model gpt gpt-4 gradio husky image-captioning langchain llm multimodal vqa internimage llama vicuna video-generation sam segment-anything click imagebind draggan
Language:Python 3220
microsoft / torchscale
Foundation Architecture for (M)LLMs
computer-vision machine-learning multimodal natural-language-processing pretrained-language-model speech-processing transformer translation
Language:Python 3119
docarray
docarray / docarray
Represent, send, store and search multimodal data
cross-modal data-structures dataclass deep-learning docarray elasticsearch fastapi machine-learning multi-modal multimodal nearest-neighbor-search nested-data neural-search protobuf pydantic pytorch qdrant semantic-search weaviate
Language:Python 3109