Giters

EvolvingLMMs-Lab / lmms-eval

Accelerating the development of large multimodal models (LMMs) with lmms-eval

Home Page:https://lmms-lab.github.io/lmms-eval-blog/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

03-25 13:08:56 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating

awzhgw opened this issue · comments

commented

03-25 13:09:00 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:01 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:02 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:382] ERROR Error LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position' in generating
03-25 13:09:03 [lmms_eval/models/llava.py:279] ERROR Error Invalid key: 3139 is out of bounds for size 2801 in getting visuals, task_type is multidocvqa_test)
Traceback (most recent call last):
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 199, in cli_evaluate
results, samples = cli_evaluate_single(args)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 283, in cli_evaluate_single
results = evaluator.simple_evaluate(
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
return fn(*args, **kwargs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate
results = evaluate(
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
return fn(*args, **kwargs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate
resps = getattr(lm, reqtype)(cloned_reqs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until
raise e
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until
visuals = [doc_to_visual0 for ids in doc_id]
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in
visuals = [doc_to_visual0 for ids in doc_id]
File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in getitem
return self._getitem(key)
File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem
pa_subtable = query_table(self._data, key, indices=self._indices)
File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table
_check_valid_index_key(key, size)
File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 3139 is out of bounds for size 2801
03-25 13:09:03 [lmms-eval/lmms_eval/main.py:213] ERROR Error during evaluation: Invalid key: 3139 is out of bounds for size 2801
Traceback (most recent call last):
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 199, in cli_evaluate
results, samples = cli_evaluate_single(args)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/main.py", line 283, in cli_evaluate_single
results = evaluator.simple_evaluate(
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
return fn(*args, **kwargs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 128, in simple_evaluate
results = evaluate(
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/utils.py", line 448, in _wrapper
return fn(*args, **kwargs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/evaluator.py", line 290, in evaluate
resps = getattr(lm, reqtype)(cloned_reqs)
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 280, in generate_until
raise e
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in generate_until
visuals = [doc_to_visual0 for ids in doc_id]
File "/export/App/training_platform/PinoModel/LLaVA/lmms-eval/lmms_eval/models/llava.py", line 277, in
visuals = [doc_to_visual0 for ids in doc_id]
File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2810, in getitem
return self._getitem(key)
File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 2794, in _getitem
pa_subtable = query_table(self._data, key, indices=self._indices)
File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 583, in query_table
_check_valid_index_key(key, size)
File "/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 3139 is out of bounds for size 2801

@ChunyuanLI @Luodian @FengLi-ust @jzhang38 @JvThunder how to resolve it ?

commented

Please stick the transformers version to 4.37.2

We would have later PR to make it compatiable with transformers > 4.37.2