mllm

There are 8 repositories under mllm topic.

microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
beit beit-3 bitnet deepnet document-ai foundation-models kosmos kosmos-1 layoutlm layoutxlm llm minilm mllm multimodal nlp pre-trained-model textdiffuser trocr unilm xlm-e
Language:Python 21819
simular-ai / Agent-S
Agent S: an open agentic framework that uses computers like a human
agent-computer-interface ai-agents computer-automation computer-use computer-use-agent cua grounding gui-agents in-context-reinforcement-learning memory mllm planning retrieval-augmented-generation
Language:Python 8091
X-PLUG / MobileAgent
Mobile-Agent: The Powerful GUI Agent Family
agent android app automation copilot gui mllm mobile mobile-agents multimodal multimodal-agent multimodal-large-language-models
Language:Python 6200
manycore-research / SpatialLM
[NeurIPS 2025] SpatialLM: Training Large Language Models for Structured Indoor Modeling
mllm point-clouds scene-understanding spatial-intelligence
Language:Python 4076
ant-research / MagicQuill
[CVPR'25] Official Implementations for Paper - MagicQuill: An Intelligent Interactive Image Editing System
aigc gradio image-editing mllm
Language:Python 3631
NExT-GPT / NExT-GPT
Code and models for ICML 2024 paper, NExT-GPT: Any-to-Any Multimodal Large Language Model
chatgpt foundation-models gpt-4 instruction-tuning large-language-models llm multi-modal-chatgpt multimodal visual-language-learning mllm
Language:Python 3579
atfortes / Awesome-LLM-Reasoning
From Chain-of-Thought prompting to OpenAI o1 and DeepSeek-R1 🍓
language-models reasoning prompt in-context-learning chatgpt chain-of-thought prompt-engineering cot awesome gpt mllm multimodal papers gpt-4o openai-o1 strawberry deepseek deepseek-r1
3409
InternLM / InternLM-XComposer
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
chatgpt visual-language-learning multi-modality foundation gpt-4 instruction-tuning mllm multimodal vision-language-model language-model large-language-model large-vision-language-model llm vision-transformer gpt supervised-finetuning
Language:Python 2900
X-PLUG / mPLUG-DocOwl
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
chart-understanding document-understanding mllm multimodal multimodal-large-language-models table-understanding
Language:Python 2258
cambrian-mllm / cambrian
Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
chatbot clip computer-vision dino instruction-tuning large-language-models llms mllm multimodal-large-language-models representation-learning
Language:Python 1957
coderonion / awesome-yolo-object-detection
🚀🚀🚀 A collection of some awesome public YOLO object detection series projects and the related object detection datasets.
cuda datasets deepseek few-shot-object-detection gui llama llm mllm object-detection object-detection-datasets open-world-object-detection qwen rknn snn spiking-neural-network tensorrt vlm yolo yolov5 yolov8
1626
bytedance / Sa2VA
Official Repo For "Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos"
computer-vision large-language-models mllm
Language:Python 1388
BAAI-DCAI / Bunny
A family of lightweight multimodal models.
chatgpt chinese english gpt-4 mllm multimodal-large-language-models vlm
Language:Python 1046
NVlabs / Eagle
Eagle: Frontier Vision-Language Models with Data-Centric Strategies
demo eagle gpt4 huggingface large-language-models llama llama3 llava llm lmm lvlm mllm nvdia
Language:Python 892
taco-group / OpenEMMA
OpenEMMA, a permissively licensed open source "reproduction" of Waymo’s EMMA model.
algorithms artificial-intelligence autonomous-car autonomous-driving autonomous-vehicles autonomy emma generative-ai machine-learning mllm networking perception transportation large-lang open-emma
Language:Python 843
CircleRadon / Osprey
[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"
mllm pixel-understanding sam visual-instruction-tuning
Language:Python 837
coderonion / awesome-llm-and-aigc
🚀🚀🚀A collection of some awesome public projects about Large Language Model(LLM), Vision Language Model(VLM), Vision Language Action(VLA), AI Generated Content(AIGC), the related Datasets and Applications.
gpt llm awesome-list llama aigc langchain datasets yolo triton cuda vlm vla deepseek qwen mllm ai4s ai4science r1 reinforcement-learning qwen3
772
LYL1015 / JarvisArt
[NeurIPS' 2025] JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent
agent image-processing large-language-models lightroom mllm vision-language-models
Language:JavaScript 681
VITA-MLLM / Woodpecker
✨✨Woodpecker: Hallucination Correction for Multimodal Large Language Models
hallucination hallucinations large-language-models llm mllm multimodal-large-language-models multimodality
Language:Python 638
taco-group / 4KAgent
[NeurIPS 2025] 4KAgent: Agentic Any Image to 4K Super-Resolution. An intelligent computer vision agent that can magically restore any image to perfect-4K!
agent agentic-ai computer-vision image-enhancement image-processing image-restoration large-language-models low-level super-resolution vision-language-models workflow llm mllm neurips neurips-2025
Language:Python 616
EvolvingLMMs-Lab / LLaVA-OneVision-1.5
Fully Open Framework for Democratized Multimodal Training
llava llm mllm qwen3 vision-language-model
Language:Python 607
FoundationVision / Groma
[ECCV2024] Grounded Multimodal Large Language Model with Localized Visual Tokenization
foundation-models grounding large-language-models llama llama2 llm mllm multimodal vision-language-model
Language:Python 578
SkyworkAI / Vitron
NeurIPS 2024 Paper: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
mllm multimodal-large-language-models segmentation
Language:Python 576
gokayfem / ComfyUI_VLM_nodes
Custom ComfyUI nodes for Vision Language Models, Large Language Models, Image to Music, Text to Music, Consistent and Random Creative Prompt Generation
comfyui custom-nodes image-captioning img2sfx img2text joytag llava llm mllm nodes phi15 siglip vlm
Language:Python 534
YingqingHe / Awesome-LLMs-meet-Multimodal-Generation
🔥🔥🔥 A curated list of papers on LLMs-based multimodal generation (image, video, 3D and audio).
aigc large-language-models large-vision-language-models llm lvlm mllm multimodal-generation multimodal-large-language-models multimodal-models multimodality text-to-3d text-to-audio text-to-image text-to-music text-to-sound text-to-speech text-to-video
Language:HTML 515
ChocoWu / Awesome-Scene-Graph-Generation
This is a repository for listing papers on scene graph generation and application.
mllm mllm-for-sg scene-graph scene-graph-generation scene-graph-to-image
491
Coobiw / MPP-LLaVA
Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
deepspeed fine-tuning mllm model-parallel multimodal-large-language-models pipeline-parallelism pretraining qwen video-language-model video-large-language-models
Language:Jupyter Notebook 477
ZebangCheng / Emotion-LLaMA
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
affective-computing instruction-tuning mllm
Language:Python 448
jingyi0000 / R1-VL
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
mllm multimodal-large-language-models r1 reasoning reinforcement-learning vision-language-model
Language:Python 432
dvlab-research / LLMGA
This project is the official implementation of 'LLMGA: Multimodal Large Language Model based Generation Assistant', ECCV2024 Oral
aigc image-design-assistant image-editing image-generation large-language-model llm mllm multi-modal
Language:Python 393
baaivision / EVE
EVE Series: Encoder-Free Vision-Language Models from BAAI
instruction-following large-language-models mllm multimodal-large-language-models vlm encoder-free-vlm llm clip vision-language-models
Language:Python 356
Atomic-man007 / Awesome_Multimodel_LLM
Awesome_Multimodel is a curated GitHub repository that provides a comprehensive collection of resources for Multimodal Large Language Models (MLLM). It covers datasets, tuning techniques, in-context learning, visual reasoning, foundational models, and more. Stay updated with the latest advancement.
chatgpt dataset gpt llm mllm multimodel nlp pretrained-models
346
VARGPT-family / VARGPT
VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model
mllm unified-model
Language:Python 345
Hon-Wong / VoRA
[Fully open] [Encoder-free MLLM] Vision as LoRA
llm lora lvlm mllm vlm
Language:Python 343
VITA-MLLM / Long-VITA
✨✨Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
long-context mllm vision-language-model
Language:Python 303
X-PLUG / Youku-mPLUG
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Pre-training Dataset and Benchmarks
benchmark chinese dataset mllm multimodal multimodal-large-language-models multimodal-pretraining video video-question-answering video-retrieval youku
Language:Python 301

mllm

microsoft / unilm

simular-ai / Agent-S

X-PLUG / MobileAgent

manycore-research / SpatialLM

ant-research / MagicQuill

NExT-GPT / NExT-GPT

atfortes / Awesome-LLM-Reasoning

InternLM / InternLM-XComposer

X-PLUG / mPLUG-DocOwl

cambrian-mllm / cambrian

coderonion / awesome-yolo-object-detection

bytedance / Sa2VA

BAAI-DCAI / Bunny

NVlabs / Eagle

taco-group / OpenEMMA

CircleRadon / Osprey

coderonion / awesome-llm-and-aigc

LYL1015 / JarvisArt

VITA-MLLM / Woodpecker

taco-group / 4KAgent

EvolvingLMMs-Lab / LLaVA-OneVision-1.5

FoundationVision / Groma

SkyworkAI / Vitron

gokayfem / ComfyUI_VLM_nodes

YingqingHe / Awesome-LLMs-meet-Multimodal-Generation

ChocoWu / Awesome-Scene-Graph-Generation

Coobiw / MPP-LLaVA

ZebangCheng / Emotion-LLaMA

jingyi0000 / R1-VL

dvlab-research / LLMGA

baaivision / EVE

Atomic-man007 / Awesome_Multimodel_LLM

VARGPT-family / VARGPT

Hon-Wong / VoRA

VITA-MLLM / Long-VITA

X-PLUG / Youku-mPLUG