ZhangYuanhan-AI

grok-1

Grok open release

llama3

The official Meta Llama 3 GitHub site

trl

Train transformer language models with reinforcement learning.

LLaVA-NeXT

Lumina-T2X

Lumina-T2X is a unified framework for Text to Any Modality Generation

LLaVA-Med

Large Language-and-Vision Assistant for Biomedicine, built towards multimodal GPT-4 level capabilities.

lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

MetaCLIP

ICLR2024 Spotlight: curation/training code, metadata, distribution and pre-trained models for MetaCLIP; CVPR 2024: MoDE: CLIP Data Experts via Clustering

ToMe

A method to increase the speed and lower the memory footprint of existing vision transformers.

VideoMamba

[ECCV2024] VideoMamba: State Space Model for Efficient Video Understanding

ring-flash-attention

Ring attention implementation with flash attention

ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

prismatic-vlms

A flexible and efficient codebase for training visually-conditioned language models (VLMs)

Video-MME

✨✨Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

ttt-lm-jax

Official JAX implementation of Learning to (Learn at Test Time): RNNs with Expressive Hidden States

scaling_on_scales

When do we not need larger vision models?

LongVA

Long Context Transfer from Language to Vision

Dataset

News: the 10k dataset is ready for download.

TimeChat

[CVPR 2024] TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

VideoRecap

Vript

LLaVA-Hound-DPO

video_captioning_datasets

Summary about Video-to-Text datasets. This repository is part of the review paper *Bridging Vision and Language from the Video-to-Text Perspective: A Comprehensive Review*

HD-VG-130M

The HD-VG-130M Dataset

PSG4D

4D Panoptic Scene Graph Generation (NeurIPS'23 Spotlight)

Genixer

(ECCV 2024) Empowering Multimodal Large Language Model as a Powerful Data Generator

MATH-V

MATH-Vision dataset and code to measure Multimodal Mathematical Reasoning capabilities.

LongVideoBench

Official Dataloader and Evaluation Scripts for LongVideoBench.

MMLongBench-Doc

Official Repository of MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations

CVRR-Evaluation-Suite

Official repository of paper titled "How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs".