Giters

open-compass / MMBench

Official Repo of "MMBench: Is Your Multi-modal Model an All-around Player?"

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The Circular Evaluation Strategy && LLM-based Choice Extractors

yyy809643573 opened this issue · comments

commented

hi, i am using the MMBench to evaluate my VLM, but i find there are some ploblems in MMBench

  1. I see that in https://github.com/open-compass/opencompass/blob/main/configs/multimodal/minigpt_4/README.md, the minigpt4 does not achieve the The Circular Evaluation Strategy and LLM-based Choice Extractors by using chatgpt，Are The Circular Evaluation Strategy and LLM-based Choice Extractors needed to achieve by ourselves？
  2. the question number of MMBench-Dev(En) is not 1164 in describtion of https://github.com/open-compass/MMBench

Looking forward to your reply，Thank you！

commented
  1. CircularEval and LLM-based Choice Extractors are implemented in eval_mmbench.py, which is not included in the inference part.
  2. The question number in the MMBench-Dev tsv is more than 1164, since it also includes CircularEval cases with choice shifted.