cjfcsjt

CuGRO

RAT

Implementation of "RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation".

webagents-step

search-agents

Code for the paper 🌳 Tree Search for Language Model Agents

MCTS-DPO

This is the repository that contains the source code for the Self-Evaluation Guided MCTS for online DPO.

digirl

Official repo for paper DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning.

IPR

Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement

Agent-Eval-Refine

Code for Paper: Autonomous Evaluation and Refinement of Digital Agents

ETO

Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents (ACL 2024 Main Conference)

AutoIF

TravelPlanner

[ICML'24 Spotlight] "TravelPlanner: A Benchmark for Real-World Planning with Language Agents"

skyvern

Automate browser-based workflows with LLMs and Computer Vision

nxtp

Object Recognition as Next Token Prediction (CVPR 2024)

groundingLMM

[CVPR 2024 🔥] Grounding Large Multimodal Model (GLaMM), the first-of-its-kind model capable of generating natural language responses that are seamlessly integrated with object segmentation masks.

Qwen-VL

The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.

honeybee

Official implementation of project Honeybee (CVPR 2024)

SoM-LLaVA

[COLM-2024] List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

vlm-evaluation

VLM Evaluation: Benchmark for VLMs, spanning text generation tasks from VQA to Captioning

cider

Pythonic wrappers for Cider/CiderD evaluation metrics. Provides CIDEr as well as CIDEr-D (CIDEr Defended) which is more robust to gaming effects. We also add the possibility to replace the original PTBTokenizer with the Spacy tekenizer (No java dependincy but slower)

VILA

VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops)

Osprey

[CVPR2024] The code for "Osprey: Pixel Understanding with Visual Instruction Tuning"

NExT-Chat

The code of the paper "NExT-Chat: An LMM for Chat, Detection and Segmentation".

bubogpt

BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs

prismatic-vlms

A flexible and efficient codebase for training visually-conditioned language models (VLMs)

LLaVA-Grounding

SeeClick

The model, data and code for the visual GUI Agent SeeClick

VisualWebBench

Evaluation framework for paper "VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?"

Long-CLIP

[ECCV 2024] official code for "Long-CLIP: Unlocking the Long-Text Capability of CLIP"

screen_annotation

The Screen Annotation dataset consists of pairs of mobile screenshots and their annotations. The annotations are in text format, and describe the UI elements present on the screen: their type, location, OCR text and a short description. It has been introduced in the paper `ScreenAI: A Vision-Language Model for UI and Infographics Understanding`.

ml-ferret