EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

https://lmms-lab.github.io/lmms-eval-blog/

EvolvingLMMs-Lab/lmms-eval Issues

Too many requests for MathVista
Updated 5 days ago
[MultiGPU Evaluation Error]: model on differencet device
Closed 5 days ago2
Extremely Low Performance with LLaVA-1.5-7B on MM-Vet Benchmark
Updated 7 days ago
Evaluation with multiple GPUs
Updated 8 days ago13
"llava_hf" is not registered
Updated 8 days ago3
Support for llava-hf/llava-v1.6-x
Updated 10 days ago3
Model will be loaded on different devices when using multiple gpus.
Closed 11 days ago6
An issue during evaluating MathVista
Updated 11 days ago1
Multinode Evaluation
Updated 13 days ago2
While this lmms-eval support V*Bench?
Updated 15 days ago
confusion about llava-next for multi-docvqa
Updated 16 days ago1
Llava1.6 Mistral ScienceQA Performance
Updated 20 days ago2
Issues when running evaluation with multiple processes
Updated 22 days ago5
How to add keywords stopping criterion during evaluation?
Closed 22 days ago1
token: False does not work in gqa.yaml
Closed 22 days ago4
Unreasonably high scores in llava_in_the_wild benchmark
Updated 23 days ago2
incomplete tasks problem
Updated 23 days ago1
How to add the new dataset?
Closed a month ago2
is there something wrong with the 000000494869.jpg in lmms-eval MME dataset ?
Updated a month ago7
different result between lmms-eval and llava-eval on text-vqa
Updated a month ago4
N/A scores when running `mmbench`
Closed a month ago10
关于CMMMU生成结果截断的问题
Updated a month ago4
OOM issue
Closed a month ago1
Remove dependency on Llava repo
Updated a month ago2
Consider using lower bounds on dependencies & using optional dependencies for developer tools
Updated a month ago3
when we can expect this support for more datasets to be completed?（Raven/NLVR2/IconQA）
Updated 2 months ago2
The discord link is expired
Closed 2 months ago2
ValueError: When passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found .
Closed 2 months ago1
lmms_eval/evaluator.py", line 83, in simple_evaluate assert tasks != [], "No tasks specified, or no tasks found. Please verify the task names." AssertionError: No tasks specified, or no tasks found. Please verify the task names.
Updated 2 months ago5
Can not using multiple batch on MMBench
Updated 2 months ago3
Logs saving json enable ensure_assiic False to support better visualize on Chinese
Updated 2 months ago
Performance about cuda version
Updated 2 months ago1
03-25 13:08:56 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
Updated 2 months ago1
Unable to reproduce llava evaluation
Closed 2 months ago7
Local dataasets config didn't work
Updated 2 months ago
How to debug in vscode
Closed 2 months ago1
upload dataset
Closed 2 months ago
How to specify dataset paths？
Updated 2 months ago12
How to load dataset in offline mode?
Closed 2 months ago2
Config will not be added to registry
Updated 2 months ago2
Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Closed 2 months ago5
/exact_match/exact_match.py. Module 'exact_match' doesn't exist on the Hugging Face Hub either.
Updated 2 months ago1
Refcoco Metric
Updated 2 months ago
Unit of data in More Results
Closed 2 months ago1
cmmmu
Closed 2 months ago3
MathVista Eval Support
Closed 2 months ago1
Typo in results sheet for llava-v1.6-vicuna-7b
Closed 2 months ago1
few-shot test error
Closed 2 months ago2
[Feature] Evaluate model with 2gpu per process.
Closed 2 months ago5
quetions about the evals
Closed 2 months ago4