Frankluox

DoraemonGPT

Official repository of DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models

FSQ-pytorch

A Pytorch Implementation of Finite Scalar Quantization

InfiAgent

RoboGen

A generative and self-guided robotic agent that endlessly propose and master new skills.

LLaMA-Pro

[ACL 2024] Progressive LLaMA with Block Expansion.

SeeAct

[ICML'24] SeeAct is a system for generalist web agents that autonomously carry out tasks on any given website, with a focus on large multimodal models (LMMs) such as GPT-4V(ision).

cosmo

Gemini-Commonsense-Evaluation

Official implementation of "Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models"

unified-io-2

MoTCoder

This is the official code repository of MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks.

vstar

PyTorch Implementation of "V* : Guided Visual Search as a Core Mechanism in Multimodal LLMs"

Large-Language-Models-play-StarCraftII

TextStarCraft2,a pure language env which support llms play starcraft2

CHOCOLATE

Code and data for the paper "Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning"

Osprey

[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"

MP5

[CVPR2024] This is the official implement of MP5

Creative-Agents

planning-as-inpainting

Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty

CM3Leon

An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images

magicoder

Magicoder: Source Code Is All You Need

IMProv

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

LVM

PCA-EVAL

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

GPT-4V-Act

AI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI

Skill-Localization-by-grafting

fit

LoRA-ViT

Low rank adaptation for Vision Transformer

FedNoisy

Benchmark for federated noisy label learning

multimodal-prompt-learning

[CVPR 2023] Official repository of paper titled "MaPLe: Multi-modal Prompt Learning".

Vision-AGI-Survey

A temporary webpage for our survey in AGI for computer vision

LabelHalluc

[AAAI 2022] Label Hallucination for Few-Shot Classification