LMMs-Lab

Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

open-r1-multimodal

A fork to add multimodal model training to open-r1

LLaVA-OneVision-1.5

Fully Open Framework for Democratized Multimodal Training

lmms-engine

A simple, unified multimodal models training engine. Lean, flexible, and built for hacking at scale.

RelateAnything

Relate Anything Model is capable of taking an image as input and utilizing SAM to identify the corresponding mask within the image.

LongVA

Long Context Transfer from Language to Vision

multimodal-search-r1

MMSearch-R1 is an end-to-end RL framework that enables LMMs to perform on-demand, multi-turn search with real-world multimodal search tools.

EgoLife

[CVPR 2025] EgoLife: Towards Egocentric Life Assistant

NEO

NEO Series: Native Vision-Language Models from First Principles

multimodal-sae

[ICCV 2025] Auto Interpretation Pipeline and many other functionalities for Multimodal SAE Analysis.

Aero-1

VideoMMMU

MGPO

High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learning

sae

A framework that allows you to apply Sparse AutoEncoder on any models

lean-runner

Deploying High-Performance Lean 4 Server in One Click

EASI

Holistic Evaluation of Multimodal LLMs on Spatial Intelligence

DeepEyes

sglang

SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

VLMEvalKit

An open-source evaluation toolkit to evaluate MLLMs on Spatial Intelligence using the EASI protocol

openevolve

Open-source implementation of AlphaEvolve

.github

agent-rl

A fork version of verl to support multi-turn tool use and many more agentic tasks.

demos

DeepseekLeanPlayground

The math library of Lean 4

DiffSynth-Studio

Enjoy the magic of Diffusion models!