OpenGenerativeAI / llm-colosseum

Benchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM

Home Page:https://huggingface.co/spaces/junior-labs/llm-colosseum

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

KeyError

jgranie opened this issue · comments

Hi, I have the following issue:
llm-colosseum/agent/robot.py", line 137, in
for button in META_INSTRUCTIONS_WITH_LOWER[combo][
for button in META_INSTRUCTIONS_WITH_LOWER[combo][
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 6
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 12

under macos m1 and local mistral, here an extract of my setup.py:
player_1=Player1(
nickname="Daddy",
#model=generate_random_model(mistral=True),
model="ollama:mistral",
),
player_2=Player2(
nickname="Baby",
#model=generate_random_model(openai=True),
model="ollama:mistral",
),

The game is displayed countdown running but both fighters stays idle.
Thanks

Hm, this is very weird.

When you run git diff, are you sure no other file than script.py was modified ?

Sorry, I forgot to reset DISABLE_LLM="False" after my first try, works fine now !
(maybe this corner case could be handled)

Cool ! Have fun :)