mllm

There are 3 repositories under mllm topic.

microsoft / unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
beit beit-3 bitnet deepnet document-ai foundation-models kosmos kosmos-1 layoutlm layoutxlm llm minilm mllm multimodal nlp pre-trained-model textdiffuser trocr unilm xlm-e
Language:Python 18452
X-PLUG / MobileAgent
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
agent gpt4v mllm mobile-agents multimodal multimodal-large-language-models multimodal-agent android app gui mobile automation copilot harmony ios
Language:Python 1863
InternLM / InternLM-XComposer
InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) excelling in free-form text-image composition and comprehension.
chatgpt foundation gpt gpt-4 instruction-tuning language-model large-language-model large-vision-language-model llm mllm multi-modality multimodal supervised-finetuning vision-language-model vision-transformer visual-language-learning
Language:Python 1700
atfortes / Awesome-LLM-Reasoning
Reasoning in Large Language Models: Papers and Resources, including Chain-of-Thought, Instruction-Tuning and Multimodality.
language-models reasoning prompt question-answering in-context-learning chatgpt chain-of-thought prompt-engineering cot awesome gpt gpt-4 mllm multimodal papers vllm
1122
X-PLUG / mPLUG-DocOwl
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
chart-understanding document-understanding mllm multimodal multimodal-large-language-models table-understanding
Language:Python 967
CircleRadon / Osprey
[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"
mllm sam visual-instruction-tuning pixel-understanding
Language:Python 695
BAAI-DCAI / Bunny
A family of lightweight multimodal models.
chatgpt chinese english gpt-4 mllm multimodal-large-language-models vlm
Language:Python 655
BradyFU / Woodpecker
✨✨Woodpecker: Hallucination Correction for Multimodal Large Language Models. The first work to correct hallucinations in MLLMs.
hallucination hallucinations large-language-models llm mllm multimodal-large-language-models multimodality
Language:Python 548
X-PLUG / Youku-mPLUG
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Pre-training Dataset and Benchmarks
benchmark chinese dataset mllm multimodal multimodal-large-language-models multimodal-pretraining video video-question-answering video-retrieval youku
Language:Python 259
X-PLUG / mPLUG-2
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video (ICML 2023)
foundation-models mllm multimodal multimodal-pretraining video image-retrieval mplug video-question-answering video-retrieval vqa
Language:Python 209
gokayfem / ComfyUI_VLM_nodes
Custom ComfyUI nodes for Vision Language Models, Large Language Models, Image to Music, Text to Music, Consistent and Random Creative Prompt Generation
comfyui custom-nodes image-captioning img2sfx img2text joytag llava llm mllm nodes phi15 siglip vlm
Language:Python 208
Atomic-man007 / Awesome_Multimodel_LLM
Awesome_Multimodel is a curated GitHub repository that provides a comprehensive collection of resources for Multimodal Large Language Models (MLLM). It covers datasets, tuning techniques, in-context learning, visual reasoning, foundational models, and more. Stay updated with the latest advancement.
chatgpt dataset gpt llm mllm multimodel nlp pretrained-models
173
360CVGroup / SEEChat
Multimodal chatbot with computer vision capabilities integrated
chatbot gpt4 mllm
Language:Python 93
FoundationVision / GenerateU
[CVPR2024] Generative Region-Language Pretraining for Open-Ended Object Detection
mllm multimodality object-detection open-vocabulary open-vocabulary-detection open-world
Language:Python 89
BAAI-DCAI / DataOptim
A collection of visual instruction tuning datasets.
llm mllm visual-instruction-tuning
Language:Python 65
Ahnsun / merlin
Merlin: Empowering Multimodal LLMs with Foresight Minds
mllm
62
X-PLUG / mPLUG-HalOwl
mPLUG-HalOwl: Multimodal Hallucination Evaluation and Mitigating
mllm multimodal-large-language-models benchmark contrastive-learning hallucinations multimodal-hallucination
Language:Python 53
VisualWebBench / VisualWebBench
Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"
computer-vision deep-learning evaluation foundation-models large-language-models large-multimodal-models llm llms machine-learning mllm multimodal multimodal-deep-learning multimodal-large-language-models natural-language-processing question-answering visual-question-answering
Language:Python 31
UCSC-VLAA / Sight-Beyond-Text
This repository includes the official implementation of our paper "Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics"
llama2 llava llm mllm vicuna vision-language ai-alignment alignment vlm
Language:Python 19
gyunggyung / OpenMLLM
Open Source + Multilingual MLLM + Fine-tuning + Distillation + More efficient models and learning + ?
efficiency efficientnet fine-tuning llm mllm multilingual open-source
Language:C 18
zzq2000 / MIKO
MIKO: Multimodal Intention Knowledge Distillation from Large Language Models for Social-Media Commonsense Discover
intention llm mllm social-media
Language:Python 17
bigai-nlco / LSTP-Chat
A Video Chat Agent with Temporal Prior
llm mllm multimodal-large-language-models spatial-temporal video-language visual-instruction-tuning
Language:Python 15
parsee-ai / parsee-datasets
Datasets, case studies and benchmarks for extracting structured information from PDFs, HTML files or images, created by the Parsee.ai team. Datasets also on Hugging Face: https://huggingface.co/parsee-ai
datasets llm mllm rag
Language:Jupyter Notebook 10
dongbzhi / AISurveyPapers
LVLM, LLM, MLLM, Multimodal Large Language Model, Large Language Model, Alignment, AI System, Survey
agent agi ai-system llm mllm survey lvlm alignment
8
isLinXu / MLLM-Research-Learn
Conducting learning and research on MLLM based on the MME rankings.
mllm
2
kassy11 / Awesome_Visually-Augmented_NLP
🖼️Latest Papers on Visually(Imagination)-Augmented NLP
awesome multimodal multimodal-deep-learning nlp dialogue-generation multimodal-fusion multimodality vision-and-language dialogue-system llm mllm multimodal-dialogue
1
xirui-li / attacks-on-LLMs
Awesome list for attacks on large language models.
attack attack-defense awsome-list defense jailbreak llm llms llms-adversarial-attack mllm vlm
1
kassy11 / Awesome_NLP_PaperList
🤖A list of PaperList of NLP related papers on Github
agents autonomous-agents awesome llm mllm multimodal multimodality natural-language-processing nlp robotics

mllm

microsoft / unilm

X-PLUG / MobileAgent

InternLM / InternLM-XComposer

atfortes / Awesome-LLM-Reasoning

X-PLUG / mPLUG-DocOwl

CircleRadon / Osprey

BAAI-DCAI / Bunny

BradyFU / Woodpecker

X-PLUG / Youku-mPLUG

X-PLUG / mPLUG-2

gokayfem / ComfyUI_VLM_nodes

Atomic-man007 / Awesome_Multimodel_LLM

360CVGroup / SEEChat

FoundationVision / GenerateU

BAAI-DCAI / DataOptim

Ahnsun / merlin

X-PLUG / mPLUG-HalOwl

VisualWebBench / VisualWebBench

UCSC-VLAA / Sight-Beyond-Text

gyunggyung / OpenMLLM

zzq2000 / MIKO

bigai-nlco / LSTP-Chat

parsee-ai / parsee-datasets

dongbzhi / AISurveyPapers

isLinXu / MLLM-Research-Learn

kassy11 / Awesome_Visually-Augmented_NLP

xirui-li / attacks-on-LLMs

kassy11 / Awesome_NLP_PaperList