AILab-CVC / SEED-Bench

(CVPR2024)A benchmark for evaluating Multimodal LLMs using multiple-choice questions.

AILab-CVC/SEED-Bench Issues

[bugs] LLaVA-Evaluation : RuntimeError: Expected all tensors to be on the same device
Updated 4 months ago
Muti-GPUs Evaluation
Updated 4 months ago
Question on how task27 generates images
Updated 4 months ago2
Request for the interface of minigpt4 and llava
Updated 5 months ago2
What is the correct way to download the video
Closed 5 months ago6
Easy way to probe result examples?
Updated 5 months ago1
a lot of data with more questions than pictures in SEED-Bench-2 level L2, is this reasonable?
Updated 5 months ago5
VLMs vs LLMs evaluation
Updated 5 months ago1
Request for the removing duplicate results
Closed 5 months ago2
Reproduce the Qwen-VL SOTAs results
Updated 5 months ago2
In-Context Example Selection Process
Updated 6 months ago1
How to download the images?
Closed 6 months ago
Support for evaluation of other VLM models like MiniGPT-4, mPLUG-Owl, Llava, and VPGTrans
Updated 8 months ago2
[Data] Could you provide a list including the files of something-something v2 which should be downloaded?
Closed 9 months ago2
This benchmark could lead to wrong conclusion.
Updated 9 months ago3
Evaluating latest version of OpenFlamingo
Closed 9 months ago1
Update for Otter-Image-MPT7B and Otter-Video
Updated 9 months ago1
Update for mPLUG-Owl
Updated 9 months ago1
Add InstructBlip Flan-T5-xl and InstructBlip Flan-T5-xxl
Updated 9 months ago1
Ask for original captions from which you generate questions
Closed 9 months ago1