epfLLM / meditron

Meditron is a suite of open-source medical Large Language Models (LLMs).

Home Page:https://huggingface.co/epfl-llm

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't benchmark on medqa

JulienVig opened this issue · comments

In the evaluation folder, running python inference.py --checkpoint mistral --checkpoint_name mistral --benchmark medqa fails with the error datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset.

I managed to fix the error by setting the MedQA benchmark attribute self.subsets = ['med_qa_en_source]

By the way, the repository requirements.txt is missing the packages wandb, scikit-learn and openai (unused though), to be able to run the benchmark suite.

commented

Thanks!