Giters

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

Home Page:https://lmms-lab.github.io/lmms-eval-blog/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

incomplete tasks problem

adrianwestmoon opened this issue · comments

commented

After installing the project, I evaluated its performance using the MME benchmark. However, it appears that the tasks I expected to see were not included in the evaluation. When I used the '--tasks list' command to display the available tasks, it became evident that only a subset of tasks had been installed. Can you assist me in resolving this issue?

[lmms-eval/lmms_eval/main.py:240] INFO Available Tasks:

  • mathverse
  • mathverse_testmini
  • mathverse_testmini_text_dominant
  • mathverse_testmini_text_lite
  • mathverse_testmini_text_only
  • mathverse_testmini_vision_dominant
  • mathverse_testmini_vision_intensive
  • mathverse_testmini_vision_only
  • mathvista
  • mathvista_test
  • mathvista_testmini
  • ocrbench
commented

If you are sure everything is under the folder and you install our packages, you might want to use lmms-eval --task list --verbosity DEBUG to see what is preventing you from loading the task.